mirror of
https://github.com/Yuyi-Oak/BlueArchiveToolkit.git
synced 2026-09-18 06:34:54 +08:00
Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 nyaKazuha
|
8fc93b8f39
|
||
|
nyaKazuha
|
e373e3fd32
|
||
|
nyaKazuha
|
7d6389806b
|
+11
-9
@@ -1,6 +1,6 @@
|
||||
# BlueArchiveToolkit 当前工作区状态
|
||||
|
||||
- **更新时间**：2026-09-04
|
||||
- **更新时间**：2026-09-06
|
||||
- **状态来源**：本地工作区盘点、代码验证和最新提交
|
||||
- **状态分支**：`experiment`
|
||||
- **最新已推送功能提交**：以当前 `git log --oneline -1` 为准
|
||||
@@ -28,13 +28,15 @@ Rust 侧官方日服资源链路已经从实验验证推进到正式入口：
|
||||
11. 官方同步会拒绝危险输出目录、路径逃逸和现有 symlink 路径组件；下载目标、`.part`、manifest、snapshot、PID、status、log 和控制锁文件不会跟随 symlink，daemon 状态类文件默认以 `0600` 权限创建。
|
||||
12. `<output>/official-version-state.json` 会明确保存当前已完成版本、正在拉取版本、上一个可用版本和失败版本；同一 app version、bundle version 和 Addressables root 的失败只保留最新一条，同一版本开始重新拉取或后续发布成功时会清理对应失败记录；`bat status` 会显示最后成功时间、下次检查时间、最后错误摘要、当前阶段、当前下载 URL 进度、版本状态摘要、最近历史失败版本和原因、结构化日志路径和轮转日志路径，人类可读输出不会把完整版本状态 JSON 内联打印。
|
||||
13. 资源导入链路已支持 CAS + `ResourceRepository` 索引写入，官方同步可用 `--import-repository` / `BAT_IMPORT_REPOSITORY=1` 在已校验 release 发布后触发导入，默认 CAS 路径为 `<output>/.cas`、SQLite 索引为 `<output>/resources.sqlite`，也可通过 `--import-cas-root`、`--import-resource-db`、`BAT_IMPORT_CAS_ROOT`、`BAT_IMPORT_RESOURCE_DB` 覆盖；`resource.index` RPC/CLI 可按类型、hash、路径模式、官方 release ID、平台、destination、bundle path、archive entry、parse status 和 TextUnit format 分页查询现有索引，release、平台、bundle path 和常用数组 metadata 过滤已下推到 SQLite，数据库不存在时返回 `available=false` 且不会创建空库；`bat doctor cas` 可只读检查既有 CAS 根目录、对象目录、元数据库文件和对象统计，不会因诊断创建空库。`Resource` metadata 已通过 `metadata_json` 兼容迁移保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式；当前/上一个/结构变化 catalog、失败 staging 复用、403/404、hash mismatch、CRC、metadata 迁移与 UnityFS 边界校验均有离线回归 fixture 或单测覆盖。
|
||||
14. 非 dry-run 官方同步在校验完成并发布后，会先对比上一完整 release 与当前 release 的 `official-download-manifest.json`，在当前 release 下写入 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`；同一 destination 只有 size 或 BLAKE3 改变才算 modified，仅 URL/CDN 根变化但内容相同不会触发解析/翻译候选。随后刷新 `official-parse-cache.json` 和 `official-textunit-index.json`，并从 Added/Modified 资源、parse cache 与 TextUnit 明细索引派生 `official-textunit-tasks.json` 和 `crowdin-textunit-queue.json`；删除资源只进入差异记录，不进入 TextUnit/Crowdin 队列。`parse.text_units` 和 `parse.errors` RPC/CLI 可按 destination、archive entry、path id、class id、field path 和 format 查询当前 release 的 TextUnit 明细与解析错误；`translation.tasks` RPC/CLI 可按 release、destination、archive entry、任务状态、parse status、TextUnit format 和 reason presence 查询离线 TextUnit 翻译任务状态与跳过/失败原因；`translation.task.update` 可回写 provider worker 状态，`translation.worker.run` 可触发 Rust provider worker 独立 claim/lease/retry 并落库 TextUnit 级译文结果，`translation.proofread` 可把汉化 workflow 标记为人工校对中；TextUnit 已包含 class id、field path、字段 offset/byte size 等可追溯定位。Crowdin provider 通过 `CROWDIN_*` 环境变量接入，mock provider 支持本地 fixture；up-to-date 轮询在已有有效解析缓存、TextUnit 明细索引和 TextUnit 队列时只读取摘要，不重复解析。官方同步报告默认 `localized_release_status=not_localized`，含义是原版资源已经发布、汉化资源未发布；受支持的 UnityFS TextAsset、TypeTree string field 和 managed-reference string field 发布成功后会写带 trace 的 `localized-patch-manifest.json`，校验 hash/size/diff/rollback 后才允许 `localized.status` 返回 `status=published`、`status_code=localized.published` 和 `localized_release_status=localized`，并可用 `localized.rollback` 显式恢复上一 release。`bat` 首次启动会在二进制所在目录释放 `.env` 配置模板（`0600`），之后每次启动自动加载（不覆盖已存在的环境变量），支持 `BAT_OUTPUT`/`BAT_LOCALIZED_OUTPUT`/`BAT_IMPORT_REPOSITORY`/`BAT_IMPORT_CAS_ROOT`/`BAT_IMPORT_RESOURCE_DB`/`BAT_STATE_DIR`/`BAT_AUTO_DISCOVER`/`BAT_WATCH`/`BAT_DAEMON`/`BAT_PROXY` 和 `BAT_TRANSLATION_*` 等键，实现编辑 `.env` 后无参启动；优先级为命令行参数 > 进程环境变量 > `.env` > 内置默认值，`BAT_SKIP_ENV_FILE=1` 可整体禁用；Redis 键为预留。daemon 任务历史持久化在 `<state-dir>/bat-tasks.json`（版本化、`0600` 原子写），重启后任务经 `task.*` 仍可查，中断任务标记 `task_interrupted`（`BAT-ERR-700005`）。官方同步报告还分别统计当前 manifest 复用、历史 release 复用、CAS 复用、网络传输字节和复用回退诊断，下载事件状态使用 `release_reused`、`cas_reused`、`downloaded` 等稳定值。
|
||||
14. 非 dry-run 官方同步在校验完成并发布后，会先对比上一完整 release 与当前 release 的 `official-download-manifest.json`，在当前 release 下写入 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`；同一 destination 只有 size 或 BLAKE3 改变才算 modified，仅 URL/CDN 根变化但内容相同不会触发解析/翻译候选。随后刷新 `official-parse-cache.json` 和 `official-textunit-index.json`，并从 Added/Modified 资源、parse cache 与 TextUnit 明细索引派生 `official-textunit-tasks.json` 和 `crowdin-textunit-queue.json`；删除资源只进入差异记录，不进入 TextUnit/Crowdin 队列。`parse.text_units` 和 `parse.errors` RPC/CLI 可按 destination、archive entry、path id、class id、field path 和 format 查询当前 release 的 TextUnit 明细与解析错误；`translation.tasks` RPC/CLI 可按 release、destination、archive entry、任务状态、parse status、TextUnit format 和 reason presence 查询离线 TextUnit 翻译任务状态与跳过/失败原因；`translation.task.update` 可回写 provider worker 状态，`translation.worker.run` 可触发 Rust provider worker 独立 claim/lease/retry 并落库 TextUnit 级译文结果，`translation.proofread` 可把汉化 workflow 标记为人工校对中；TextUnit 已包含 class id、field path、字段 offset/byte size 等可追溯定位。Crowdin provider 通过 `CROWDIN_*` 环境变量接入，mock provider 支持本地 fixture；up-to-date 轮询在已有有效解析缓存、TextUnit 明细索引和 TextUnit 队列时只读取摘要，不重复解析。官方同步报告默认 `localized_release_status=not_localized`，含义是原版资源已经发布、汉化资源未发布；受支持的 UnityFS TextAsset、TypeTree string field 和 managed-reference string field 发布成功后会写带 trace 的 `localized-patch-manifest.json`，校验 hash/size/diff/rollback 后才允许 `localized.status` 返回 `status=published`、`status_code=localized.published` 和 `localized_release_status=localized`，并可用 `localized.rollback` 显式恢复上一 release。`bat` 首次启动会在二进制所在目录释放 `config.toml.example` 配置模板（`0600`），`config.toml` 存在且 Unix 权限为 `0600` 或更严格时读取并使用它；`config.toml` 不存在时仅保留模板，不自动读取 example，运行时继续使用环境变量和内置默认值。优先级为命令行参数 > 进程环境变量 > `config.toml` > 内置默认值，`BAT_SKIP_ENV_FILE` 已废弃且不再影响启动；Redis 键为预留。daemon 任务历史持久化在 `<state-dir>/bat-tasks.json`（版本化、`0600` 原子写），重启后任务经 `task.*` 仍可查，中断任务标记 `task_interrupted`（`BAT-ERR-700005`）。官方同步报告还分别统计当前 manifest 复用、历史 release 复用、CAS 复用、网络传输字节和复用回退诊断，下载事件状态使用 `release_reused`、`cas_reused`、`downloaded` 等稳定值。
|
||||
|
||||
15. Rust `bat` 已提供 `res` / `parse` / `i18n` 工作流入口：支持资源拉取、解析刷新、可再生解析缓存清理、离线翻译工作台、人工文本查看/修改/清空、工作台发布前校验、有限 TextAsset 汉化发布、人工校对状态标记、既有 patch 能力的批量重打包、单次/限定次数/周期执行和版本化 schedule CRUD。`translation.worker.run` 已接入 provider worker：默认并发 8、范围 `1..=256`，每个 worker 独立 claim 下一项任务并落库 lease、失败分类、重试计划和 TextUnit 译文结果。schedule 查询现在按一级工作流过滤，删除/执行会校验作用域，单轮执行可限制计划数；schedule CRUD、翻译任务查询/交接视图、翻译任务状态回写、provider worker 触发和 `translation.proofread` 状态标记已通过 `bat.sock` 的 RPC 以及 `bat-api` 的鉴权管理接口暴露，dashboard 不维护第二套状态。`bat-api` 已提供内嵌 dashboard MVP，静态资产由 Go embed 暴露在 `/admin/dashboard/`，页面直接调用已有鉴权接口控制资源、调度、翻译、任务、日志、parse TextUnit 查询和 localized 发布/回滚。该工作流只编排已有解析和 patch 能力，不扩大解析器覆盖；完整 AssetBundle 重打包和完整 Web 协作后台仍是后续工作。真实官方网络全量拉取 smoke 已固化，真实大文件与运行报告默认在 `/tmp` 隔离目录，不纳入 Git。Go 细节见 `docs/reports/GO_STATUS.md`。
|
||||
|
||||
当前翻译交接还包括 `translation-tasks.sqlite` 和版本化 `translation-handoff.json`；
|
||||
当前翻译交接还包括 `translation-tasks.sqlite` 和版本化 `translation-handoff.json`；跨 release 的 Translation Memory V1 位于 `<output>/translation-memory.sqlite`，不放在 `versions/<id>` 或 release task 库中；
|
||||
`translation.tasks` 查询单项 worker 状态，`translation.handoff` 查询完整
|
||||
job/unit/provider run 状态。当前下载实现使用默认 8 个独立 worker，完成后动态领取
|
||||
job/unit/provider run 状态；`translation.memory.summary/query/confirm` 提供
|
||||
Rust-owned TM 的摘要、source/context 查询和显式 trusted 确认，`bat-api` 仅作
|
||||
typed 管理转发。当前下载实现使用默认 8 个独立 worker，完成后动态领取
|
||||
任务，最终资源报告按 pull plan 顺序输出。
|
||||
|
||||
---
|
||||
@@ -147,7 +149,7 @@ job/unit/provider run 状态。当前下载实现使用默认 8 个独立 worker
|
||||
|
||||
待完成：
|
||||
|
||||
- 基于已接入的 `translation.worker.run` 继续推进翻译记忆、完整 Patch 构建/rollback；继续扩展更丰富的 TextUnit 查询和通用 Patch 发布资源视图。
|
||||
- 基于已接入的 `translation.worker.run` 继续推进 Glossary、完整 Patch 构建/rollback；继续扩展更丰富的 TextUnit/TM 查询和通用 Patch 发布资源视图。
|
||||
- 真实线上全量下载 smoke 已固化为 `scripts/official-full-pull-smoke.sh` 和 `make official-smoke`；实际运行报告由脚本写入隔离输出目录。
|
||||
- 增加更多权限和极端文件系统场景测试。
|
||||
|
||||
@@ -270,7 +272,7 @@ cargo run -p bat-infrastructure --bin bat -- \
|
||||
--watch
|
||||
```
|
||||
|
||||
资源 HTTP bootstrap / 只读分发入口是 Go `cmd/bat-api`。生产拓扑下它与 Rust `bat` 同环境运行，经 `bat.sock` RPC 获取当前 `resource_root`，不在配置里写死资源目录；本地开发不能全量跑 `bat` 时用 fixture 和 Go 门禁验证。`internal/api/testdata/contract/` 已固化来自 Rust 输出并经归一化的 `catalog.status`、`resource.manifest` 和 `official-sync-snapshot.json` contract fixture，Go mirror 测试会防止字段名、null 语义和 `game_main_config_bootstrap` 再次漂移。`bat-api` 已补 launcher 资源引导兼容端点、玩家-facing HTTP 控制面和鉴权调度/translation 管理接口（token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI、管理控制白名单；`reload` / `refresh` / `restart` / `sync` / `verify` / `repair` / `catalog-refresh`、`schedule.*`、`task.*` 查询/取消、`daemon.logs`、`parse.*` 查询、`translation.tasks` / `translation.handoff` 查询、`translation.task.update`、`translation.worker.run`、`translation.proofread`、`localized.publish` 和 `localized.rollback` 可经 dashboard 转发），响应只来自已发布 snapshot/RPC，不提供官方账号登录、游戏网关协议或完整 package update manifest。
|
||||
资源 HTTP bootstrap / 只读分发入口是 Go `cmd/bat-api`。生产拓扑下它与 Rust `bat` 同环境运行，经 `bat.sock` RPC 获取当前 `resource_root`，不在配置里写死资源目录；本地开发不能全量跑 `bat` 时用 fixture 和 Go 门禁验证。`internal/api/testdata/contract/` 已固化来自 Rust 输出并经归一化的 `catalog.status`、`resource.manifest` 和 `official-sync-snapshot.json` contract fixture，Go mirror 测试会防止字段名、null 语义和 `game_main_config_bootstrap` 再次漂移；TM 另有 Rust/Go 字段镜像测试覆盖 match、trust、translated text 和 provenance。`bat-api` 已补 launcher 资源引导兼容端点、玩家-facing HTTP 控制面和鉴权调度/translation/TM 管理接口（token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI、管理控制白名单；`reload` / `refresh` / `restart` / `sync` / `verify` / `repair` / `catalog-refresh`、`schedule.*`、`task.*` 查询/取消、`daemon.logs`、`parse.*` 查询、`translation.tasks` / `translation.handoff` 查询、`translation.task.update`、`translation.worker.run`、`translation.proofread`、`translation.memory.summary/query/confirm`、`localized.publish` 和 `localized.rollback` 可经 dashboard/API 转发），响应只来自已发布 snapshot/RPC，不提供官方账号登录、游戏网关协议或完整 package update manifest。
|
||||
|
||||
生产要求：
|
||||
|
||||
@@ -297,7 +299,7 @@ Issue 状态不作为本地实现状态的权威来源；本次复核未把远
|
||||
|
||||
1. 继续复杂 AssetBundle：真实样本、复杂字段解析和发布级重打包。
|
||||
2. 继续通用 Patch：manifest 驱动、双 release 查询和清理策略。
|
||||
3. 继续资源查询和翻译基础设施：更丰富的查询、Translation Memory、Glossary 和 Provider
|
||||
3. 继续资源查询和翻译基础设施：更丰富的查询、Glossary 和 Provider
|
||||
扩展体系。
|
||||
4. 在资源和翻译契约稳定后推进完整 Web 协作后台和完整游戏业务 API。
|
||||
|
||||
@@ -308,6 +310,6 @@ Issue 状态不作为本地实现状态的权威来源；本次复核未把远
|
||||
HTTP 控制面、launcher 资源引导兼容、RPC 周期刷新/诊断、readiness、内嵌 dashboard
|
||||
和 `backendrpc` 可用；CAS 用户级导入、TextUnit 明细索引/查询、增量离线队列、
|
||||
通用 Binary/JSON/Text Patch 基础和受支持 localized patch 发布/rollback 可用；
|
||||
完整 AssetBundle 重打包、完整 Web 协作后台、翻译记忆和通用 manifest 发布未完成。
|
||||
- **下一工程里程碑**：复杂 AssetBundle 解析和重打包、翻译记忆、通用 manifest Patch
|
||||
完整 AssetBundle 重打包、完整 Web 协作后台、Glossary、模糊 TM 匹配和通用 manifest 发布未完成。
|
||||
- **下一工程里程碑**：复杂 AssetBundle 解析和重打包、Glossary、通用 manifest Patch
|
||||
构建，以及真实官方资源长期运行验证。
|
||||
|
||||
+15
-16
@@ -32,11 +32,11 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付
|
||||
4. `bat-cas-engine` 已完成 CAS V1：原子写入、BLAKE3 Hash、SQLite 引用计数、GC、并发测试、损坏检测。
|
||||
5. `bat-infrastructure` 已改为 CAS 仓储适配层，不再重复实现对象存储。
|
||||
6. `bat-infrastructure` 已提供官方资源 pull/update 服务，正式入口是 Rust binary `bat`。
|
||||
7. `bat` 支持 `--auto-discover`、`--watch`、`--daemon`、默认 1 小时间隔、本地 manifest audit/repair、官方 seed `.hash` 校验、snapshot/cache，以及基于 Unix socket JSON-RPC 的 live control/backend 方法（`daemon.status/logs/stop/restart/reload/refresh/doctor`、`resource.sync/verify/repair/state/manifest/list/index`、`parse.status/text_units/errors`、`localized.status`、`catalog.*`、`task.*`）；`daemon.restart` 通过 Rust lifecycle controller 复用 CLI restart 路径，`clean-stable` 仍由 CLI 侧按进程生命周期显式执行。
|
||||
7. `bat` 支持 `--auto-discover`、`--watch`、`--daemon`、默认 1 小时间隔、本地 manifest audit/repair、官方 seed `.hash` 校验、snapshot/cache，以及基于 Unix socket JSON-RPC 的 live control/backend 方法（`daemon.status/logs/stop/restart/reload/refresh/doctor`、`resource.sync/verify/repair/state/manifest/list/index`、`parse.status/text_units/errors`、`translation.*`、`localized.status`、`catalog.*`、`task.*`）；`daemon.restart` 通过 Rust lifecycle controller 复用 CLI restart 路径，`clean-stable` 仍由 CLI 侧按进程生命周期显式执行。
|
||||
8. `bat-ffi` 已提供 Manifest inspect 和官方 sync plan 的可选无状态粗粒度 JSON C ABI helper。
|
||||
9. 官方原版资源默认发布到 `./bat-resources`，汉化产物默认发布到独立的 `./bat-localized`；当前官方同步报告会标记 `localized_release_status=not_localized`，表示原版资源已发布、汉化资源未发布；`translation.proofread` 可把汉化 workflow 标记为人工校对中，但不会覆盖已发布的汉化 release。
|
||||
10. 官方同步校验完成并发布新 release 后会生成 `official-resource-changes.json`、`crowdin-translation-handoff.json`、`official-parse-cache.json`、`official-textunit-index.json`、`official-textunit-tasks.json` 和 `crowdin-textunit-queue.json`，用 Added/Modified 资源驱动后续解析/翻译增量；up-to-date 轮询在已有有效缓存、TextUnit 明细索引和队列时只读取摘要，不重复解析。
|
||||
11. Rust `bat` 已提供 `res` / `parse` / `i18n` 工作流：单次/限定次数/周期执行、版本化 schedule CRUD 与作用域过滤、解析缓存清理、翻译工作台校验、离线翻译工作台、人工文本查看/修改/清空、翻译任务状态回写、人工校对状态标记、既有 patch 能力的批量重打包和独立汉化 release 发布；`translation.worker.run` 已接入 provider worker，默认并发 8、范围 `1..=256`，每个 worker 独立 claim 下一项任务并落库 lease、失败分类、重试计划和 TextUnit 译文结果；`bat-api` 已提供内嵌 dashboard MVP，直接调用已有鉴权接口控制资源、调度、任务、日志、parse、翻译和 localized 发布/回滚；schedule CRUD、`translation.tasks` / `translation.handoff`、`translation.task.update`、`translation.worker.run` 和 `translation.proofread` 已经通过 `bat.sock` 和 `bat-api` 管理接口暴露，dashboard 不维护第二套状态；新解析覆盖仍需真实 fixture 和回归验收。
|
||||
11. Rust `bat` 已提供 `res` / `parse` / `i18n` 工作流：单次/限定次数/周期执行、版本化 schedule CRUD 与作用域过滤、解析缓存清理、翻译工作台校验、离线翻译工作台、人工文本查看/修改/清空、翻译任务状态回写、人工校对状态标记、既有 patch 能力的批量重打包和独立汉化 release 发布；`translation.worker.run` 已接入 provider worker，默认并发 8、范围 `1..=256`，每个 worker 独立 claim 下一项任务并落库 lease、失败分类、重试计划和 TextUnit 译文结果；`bat-api` 已提供内嵌 dashboard MVP，直接调用已有鉴权接口控制资源、调度、任务、日志、parse、翻译、TM 和 localized 发布/回滚；schedule CRUD、`translation.tasks` / `translation.handoff`、`translation.task.update`、`translation.worker.run`、`translation.proofread` 和 `translation.memory.*` 已经通过 `bat.sock` 和 `bat-api` 管理接口暴露，dashboard 不维护第二套状态；新解析覆盖仍需真实 fixture 和回归验收。
|
||||
12. 文档已整理：根目录保留入口文档，历史报告进入 `docs/reports/historical/`，误嵌套的 `docs/docs` 已合并。
|
||||
|
||||
### 仍是骨架或占位
|
||||
@@ -45,8 +45,8 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付
|
||||
2. `bat-patch` 已具备确定性 Binary hunk diff/apply、RFC 6902 JSON Patch apply、UTF-8 Text Patch、通用 Patch manifest、BLAKE3/size 完整性校验和 rollback 元数据；文件级 `patch.apply` RPC / `patch-apply` CLI 与 UnityFS TextAsset / TypeTree string / TypeTree 语义字段写入入口已开放，`bat-assetbundle` + `LocalizedPatchService` 已完成受支持 TextUnit 到 localized patch manifest、独立 staging、发布和 rollback 闭环，通用 manifest 发布与整体 AssetBundle 重打包仍后置。
|
||||
3. Go 侧边界已确定（见 `docs/reports/GO_STATUS.md`）：同步/运维命令行 = Rust `bat`；资源分发和内嵌 dashboard = `cmd/bat-api` MVP；`internal/backendrpc` 完成；`cmd/bat` 仅为试验（`bin/bat-go`）。完整游戏业务 API / 完整 Web 协作后台 / SDK 仍未完成。
|
||||
4. Addressables parser 已覆盖当前真实形态 fixture/golden，但还不是完整 Unity Addressables/SBP catalog 兼容层。
|
||||
5. 官方同步结果可配置为发布后自动导入 CAS + ResourceRepository，并通过 `resource.index` RPC/CLI 查询；Resource metadata 已保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 摘要，资源级查询已覆盖 release、平台、destination、archive entry、parse status 和 TextUnit format；单条 TextUnit 明细和解析错误已持久化到 `official-textunit-index.json`，可通过 `parse.text_units` / `parse.errors` 查询；离线 TextUnit 翻译任务状态和跳过/失败原因可通过 `translation.tasks` 查询，`translation.task.update` 已提供 worker 状态回写 contract，`translation.worker.run` 已提供真实 provider worker 触发、lease/retry 和结果落库 contract，`translation.proofread` 已提供汉化 workflow 人工校对标记 contract。
|
||||
6. 受支持汉化 Patch 发布已具备 UnityFS TextAsset、TypeTree string field 和 managed-reference string field 的 manifest/apply/rollback/完整性校验和 `localized.status` 严格校验；真实 provider worker 已接入，翻译记忆到完整汉化文件集合的构建仍未完成。
|
||||
5. 官方同步结果可配置为发布后自动导入 CAS + ResourceRepository，并通过 `resource.index` RPC/CLI 查询；Resource metadata 已保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 摘要，资源级查询已覆盖 release、平台、destination、archive entry、parse status 和 TextUnit format；单条 TextUnit 明细和解析错误已持久化到 `official-textunit-index.json`，可通过 `parse.text_units` / `parse.errors` 查询；离线 TextUnit 翻译任务状态和跳过/失败原因可通过 `translation.tasks` 查询，`translation.task.update` 已提供 worker 状态回写 contract，`translation.worker.run` 已提供真实 provider worker 触发、lease/retry 和结果落库 contract，`translation.proofread` 已提供汉化 workflow 人工校对标记 contract，`translation.memory.*` 已提供 Rust-owned TM 摘要、raw source/context 查询、provenance 和显式 confirm contract，Go 侧仅代理。
|
||||
6. 受支持汉化 Patch 发布已具备 UnityFS TextAsset、TypeTree string field 和 managed-reference string field 的 manifest/apply/rollback/完整性校验和 `localized.status` 严格校验；真实 provider worker 与项目级 Translation Memory V1 已接入，翻译记忆到完整汉化文件集合的构建仍未完成。
|
||||
7. 真实官方网络全量下载 smoke 已固化为可重复脚本和 runbook；真实运行记录处于长期运行测试阶段，报告待后续提供。
|
||||
8. 内嵌 dashboard MVP 已实现；完整 Web 协作后台、数据库迁移、插件加载机制尚未实现；`bat-api` 资源 bootstrap/分发/OpenAPI/管理控制面已通过 `/openapi.yaml` 提供，完整游戏业务 API 的 OpenAPI 仍未完成。
|
||||
9. 原 Git 历史未恢复；当前仓库以新初始化基线为准。
|
||||
@@ -169,7 +169,7 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付
|
||||
|
||||
**目标**：能够获取、解析和同步 Blue Archive 资源清单。
|
||||
|
||||
**当前状态**：部分完成。Rust 官方日服资源同步链路已经具备正式 one-shot 和 `--watch` 常驻入口；`bat-api` 资源 bootstrap/分发入口已落地，CAS + ResourceRepository 导入、历史 release/CAS 复用已可用，但完整解析覆盖、丰富查询/翻译记忆和真实线上 smoke 仍待完成。
|
||||
**当前状态**：部分完成。Rust 官方日服资源同步链路已经具备正式 one-shot 和 `--watch` 常驻入口；`bat-api` 资源 bootstrap/分发入口已落地，CAS + ResourceRepository 导入、历史 release/CAS 复用和 Translation Memory V1 已可用，但完整解析覆盖、丰富查询扩展和真实线上 smoke 仍待完成。
|
||||
|
||||
交付物：
|
||||
|
||||
@@ -181,7 +181,7 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付
|
||||
6. Go 用户级 `sync`、`manifest inspect`、`cache status`：**未完成**。Rust `bat`
|
||||
是当前正式资源同步 CLI，`bat --json` 是其机器输出形态；`bat-ffi` 只提供可选
|
||||
兼容用的 Manifest inspect 和 sync plan JSON helper。
|
||||
7. 下载结果写入 CAS + ResourceRepository：**基础能力可用，查询面仍部分完成**。CAS 和 SQLite ResourceRepository 已存在，官方同步入口可用 `--import-repository` / `BAT_IMPORT_REPOSITORY=1` 在已校验 release 发布后导入；`resource.index` 可按资源级 release、平台、destination、bundle path、archive entry、parse status 和 TextUnit format 查询现有索引和资源 metadata，常用 metadata 过滤已下推到 SQLite；`bat doctor cas` 可只读诊断既有 CAS 根目录、对象目录、元数据库文件和对象统计；`parse.text_units` / `parse.errors` 可查询当前 release 的 TextUnit 明细与解析错误；`translation.tasks` 可查询离线 TextUnit 翻译任务状态、跳过/失败原因和 worker 结果。剩余工作是更丰富的 TextUnit 查询、翻译记忆查询和通用 Patch 发布所需资源视图。
|
||||
7. 下载结果写入 CAS + ResourceRepository：**基础能力可用，查询面仍部分完成**。CAS 和 SQLite ResourceRepository 已存在，官方同步入口可用 `--import-repository` / `BAT_IMPORT_REPOSITORY=1` 在已校验 release 发布后导入；`resource.index` 可按资源级 release、平台、destination、bundle path、archive entry、parse status 和 TextUnit format 查询现有索引和资源 metadata，常用 metadata 过滤已下推到 SQLite；`bat doctor cas` 可只读诊断既有 CAS 根目录、对象目录、元数据库文件和对象统计；`parse.text_units` / `parse.errors` 可查询当前 release 的 TextUnit 明细与解析错误；`translation.tasks` 和 `translation.memory.*` 可查询离线 TextUnit 与项目级 TM。剩余工作是更丰富的 TextUnit/TM 查询和通用 Patch 发布所需资源视图。
|
||||
8. Linux 生产同步不依赖已安装官方启动器：**已完成当前 Rust 入口**。`--auto-discover` 只使用官方 HTTP metadata 和临时目录解析 `GameMainConfig`。
|
||||
9. 真实官方网络全量下载 smoke test：**命令已固化**。`scripts/official-full-pull-smoke.sh` / `make official-smoke` 已固化 dry-run、首次下载、二次 up-to-date 和本地损坏 repair 的可重复流程；真实运行处于长期运行测试阶段，报告待后续提供。
|
||||
10. 官方发布后的增量 handoff 与解析缓存：**已完成基础入口**。新 release 发布后先生成 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`，新增+变更资源进入解析/翻译候选；`official-parse-cache.json` 基于下载 manifest 覆盖直接 UnityFS bundle、zip 内 UnityFS 条目和非候选资源记录；随后生成 `official-textunit-index.json`、`official-textunit-tasks.json` 与 `crowdin-textunit-queue.json`，本地文件未变化且缓存/索引有效时跳过重复解析。
|
||||
@@ -260,11 +260,10 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付
|
||||
|
||||
交付物：
|
||||
|
||||
1. PostgreSQL schema：source_text、translation、translation_memory、glossary、review、history。
|
||||
2. 实现精确匹配、模糊匹配、上下文匹配。
|
||||
3. 实现术语优先级、别名、分类、冲突检测和审核状态。
|
||||
4. 实现导入导出和版本历史。
|
||||
5. 实现 `translate memory`、`glossary` CLI 子命令。
|
||||
1. Translation Memory V1 已使用项目级 SQLite schema：source raw/hash、translation、完整 context、candidate/trusted 和 provenance。
|
||||
2. 已实现 raw source + 完整 context exact match；模糊匹配、Glossary 联动和完整导入导出仍待实现。
|
||||
3. 已实现显式 per-record confirm；术语优先级、别名、分类、冲突检测和审核队列仍待实现。
|
||||
4. 已实现 `bat i18n memory summary|query|confirm` 与对应 Rust RPC。
|
||||
|
||||
验收标准：
|
||||
|
||||
@@ -388,7 +387,7 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付
|
||||
|
||||
## 5. 推荐执行顺序
|
||||
|
||||
近期不要把内嵌 dashboard MVP 扩成完整协作后台或过早扩展 AI Provider。项目当前的真实瓶颈仍是翻译记忆、通用 manifest 发布、复杂 AssetBundle 重打包和真实官方资源长期运行验证。
|
||||
近期不要把内嵌 dashboard MVP 扩成完整协作后台或过早扩展 AI Provider。项目当前的真实瓶颈仍是 Glossary、通用 manifest 发布、复杂 AssetBundle 重打包和真实官方资源长期运行验证。
|
||||
|
||||
建议顺序：
|
||||
|
||||
@@ -406,8 +405,8 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付
|
||||
`docs/reports/GO_STATUS.md`：
|
||||
|
||||
1. 继续 Addressables 结构变体与 UnityFS 复杂对象能力。
|
||||
2. 基于 `translation.worker.run` 继续推进翻译记忆和 Patch 构建。
|
||||
3. 继续扩展资源库剩余查询面：更丰富的 TextUnit 查询、翻译记忆查询和通用 Patch 发布所需资源视图。
|
||||
2. 基于 `translation.worker.run` 继续推进 Glossary 和 Patch 构建。
|
||||
3. 继续扩展资源库剩余查询面：更丰富的 TextUnit/TM 查询和通用 Patch 发布所需资源视图。
|
||||
4. 在隔离环境执行真实官方网络长期运行 smoke，并保留运行报告。
|
||||
|
||||
---
|
||||
@@ -462,9 +461,9 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付
|
||||
|
||||
按最终目标计算，当前总体完成度不固定写单一百分比，以模块状态、源码、测试和契约为准。
|
||||
|
||||
已完成的是稳定基线、架构骨架、部分接口、CAS V1、Rust 官方资源同步闭环、可配置 CAS/ResourceRepository 导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、provider worker、通用 Binary/JSON/Text Patch 基础、受支持 localized patch 发布/rollback，以及 Go `bat-api` 资源分发、内嵌 dashboard 和同机 live 联调。下一阶段的关键是翻译记忆、通用 manifest 发布、复杂 AssetBundle 解析/重打包和官方资源长期运行报告。
|
||||
已完成的是稳定基线、架构骨架、部分接口、CAS V1、Rust 官方资源同步闭环、可配置 CAS/ResourceRepository 导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、provider worker、Translation Memory V1、通用 Binary/JSON/Text Patch 基础、受支持 localized patch 发布/rollback，以及 Go `bat-api` 资源分发、内嵌 dashboard 和同机 live 联调。下一阶段的关键是 TM 扩展、Glossary、通用 manifest 发布、复杂 AssetBundle 解析/重打包和官方资源长期运行报告。
|
||||
|
||||
---
|
||||
|
||||
- **下一份应补充的验证材料**：真实官方网络 smoke 运行记录
|
||||
- **下一项工程任务**：推进翻译记忆、通用 manifest Patch 构建、复杂 AssetBundle 解析，并持续执行官方资源长期运行 smoke。
|
||||
- **下一项工程任务**：推进 TM 扩展、Glossary、通用 manifest Patch 构建、复杂 AssetBundle 解析，并持续执行官方资源长期运行 smoke。
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**BlueArchiveToolkit** 是一个面向长期维护的 Blue Archive 资源管理、解析、翻译和补丁工具套件。
|
||||
|
||||
当前仓库仍不是完整产品，但 Rust 侧已经具备一条可运行的官方日服资源同步链路：可以在 Linux 上通过官方 HTTP metadata 自动发现资源入口，拉取 Windows + Android 官方资源，保存同步 snapshot，校验本地下载清单，并用近乎全自动的 `--watch` / `--daemon` 常驻更新。Go module 名为 `bat-api`：正式 Go 入口是资源 bootstrap + 分发服务 `cmd/bat-api`（与 Rust `bat` 同环境运行，经 `bat.sock` RPC 周期发现 release 和 `resource_root`，提供 `/v1/bootstrap`、server-info 改写、CDN path 只读分发和内嵌管理 dashboard）；`internal/backendrpc` 为 RPC client；`cmd/bat` 仅为试验骨架（产物 `bin/bat-go`，不是产品 CLI）。边界与进度见 [`docs/reports/GO_STATUS.md`](docs/reports/GO_STATUS.md)。完整游戏业务 API、完整 Web 协作后台、复杂 AssetBundle 重打包、翻译记忆和通用 Patch 发布仍在后续阶段。
|
||||
当前仓库仍不是完整产品，但 Rust 侧已经具备一条可运行的官方日服资源同步链路：可以在 Linux 上通过官方 HTTP metadata 自动发现资源入口，拉取 Windows + Android 官方资源，保存同步 snapshot，校验本地下载清单，并用近乎全自动的 `--watch` / `--daemon` 常驻更新。Go module 名为 `bat-api`：正式 Go 入口是资源 bootstrap + 分发服务 `cmd/bat-api`（与 Rust `bat` 同环境运行，经 `bat.sock` RPC 周期发现 release 和 `resource_root`，提供 `/v1/bootstrap`、server-info 改写、CDN path 只读分发和内嵌管理 dashboard）；`internal/backendrpc` 为 RPC client；`cmd/bat` 仅为试验骨架（产物 `bin/bat-go`，不是产品 CLI）。边界与进度见 [`docs/reports/GO_STATUS.md`](docs/reports/GO_STATUS.md)。完整游戏业务 API、完整 Web 协作后台、复杂 AssetBundle 重打包和通用 Patch 发布仍在后续阶段。
|
||||
|
||||
---
|
||||
|
||||
@@ -13,14 +13,14 @@
|
||||
- `bat-adapters` Unity、Manifest、Client 集成框架，以及当前真实形态 Addressables catalog 解析覆盖，含 `m_Crc` 提取和 UnityFS 解包/TextAsset 提取基础校验。
|
||||
- `bat-cas-engine` CAS V1：原子写入、BLAKE3 校验、引用计数、GC、并发写入测试、损坏检测。
|
||||
- `bat-infrastructure` CAS 适配层、SQLite Resource Repository、资源导入服务、官方资源 pull/update 服务。
|
||||
- `bat`：官方资源自动发现、全量拉取、原子发布到 `current -> versions/<id>`、本地 manifest audit/repair、`.part` 断点续传、403/404/5xx 分类重试、指数退避、默认并发 8（可配置 `1..=256`，report 按 plan 顺序、进度按完成数单调上报）、已发布历史 release 与 CAS 复用、下载 quarantine 诊断、ZIP 结构校验、官方 seed `.hash` 校验、snapshot/cache、版本化 `official-launcher-bootstrap.json`、`--watch` 常驻更新、`--daemon` 后台运行，以及 Unix socket JSON-RPC live control/backend 方法（`daemon.*`、`resource.*`、`parse.*`、`translation.tasks/handoff/task.update`、`localized.status`、`catalog.*`、`task.*`、`patch.apply`、`unityfs.patch_*`）。
|
||||
- `bat`：官方资源自动发现、全量拉取、原子发布到 `current -> versions/<id>`、本地 manifest audit/repair、`.part` 断点续传、403/404/5xx 分类重试、指数退避、默认并发 8（可配置 `1..=256`，report 按 plan 顺序、进度按完成数单调上报）、已发布历史 release 与 CAS 复用、下载 quarantine 诊断、ZIP 结构校验、官方 seed `.hash` 校验、snapshot/cache、版本化 `official-launcher-bootstrap.json`、`--watch` 常驻更新、`--daemon` 后台运行，以及 Unix socket JSON-RPC live control/backend 方法（`daemon.*`、`resource.*`、`parse.*`、`translation.tasks/handoff/task.update/worker.run`、`translation.memory.*`、`localized.status`、`catalog.*`、`task.*`、`patch.apply`、`unityfs.patch_*`）。
|
||||
- `internal/backendrpc`：Go 侧 typed Unix socket JSON-RPC client，是 `bat-api` 调用 Rust daemon 的默认路径。
|
||||
- `cmd/bat-api`：资源 bootstrap + 分发 HTTP MVP（G-009）；`/v1/bootstrap` 和 `/v1/launcher/bootstrap` 组织 `bat` 已发布 release 的启动前资源入口，launcher 形状兼容端点仅输出资源 metadata / GameMainConfig 引导，`/healthz` 暴露 RPC refresh 诊断，`/readyz` 做 release readiness，CDN path 支持 `GET`/`HEAD`/`Range`、ETag、Last-Modified 和缓存头；玩家-facing 控制面已具备 token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI、管理控制白名单、task/log/parse/translation admin 查询控制入口和无构建内嵌 dashboard；`.env` 配置端口/RPC socket/刷新周期；生产资源根和长期状态来自 RPC，不负责自动拉取。
|
||||
- `cmd/bat-api`：资源 bootstrap + 分发 HTTP MVP（G-009）；`/v1/bootstrap` 和 `/v1/launcher/bootstrap` 组织 `bat` 已发布 release 的启动前资源入口，launcher 形状兼容端点仅输出资源 metadata / GameMainConfig 引导，`/healthz` 暴露 RPC refresh 诊断，`/readyz` 做 release readiness，CDN path 支持 `GET`/`HEAD`/`Range`、ETag、Last-Modified 和缓存头；玩家-facing 控制面已具备 token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI、管理控制白名单、task/log/parse/translation/TM admin 查询控制入口和无构建内嵌 dashboard；`.env` 配置端口/RPC socket/刷新周期；生产资源根和长期状态来自 RPC，不负责自动拉取。
|
||||
- Go 边界权威说明：[`docs/reports/GO_STATUS.md`](docs/reports/GO_STATUS.md)（同步 CLI = Rust `bat`）。
|
||||
- 官方同步会维护 `<output>/official-version-state.json`，明确记录当前已完成版本、正在拉取版本、上一个可用版本和失败版本。
|
||||
- 资源导入链路可配置为在官方 release 发布后写入 CAS + `ResourceRepository`，资源 metadata 会记录 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式，TextAsset/Table/Media 会按类型分类索引；`resource.index` RPC/CLI 可按类型、hash、路径模式、官方 release ID、平台、destination、bundle path、archive entry、parse status 和 TextUnit format 分页查询索引，常用 metadata 过滤会下推到 SQLite；历史 release 复用会重新校验 size、BLAKE3 和 ZIP 结构，失败时按历史 release、CAS、网络顺序回退，CAS 引用记录在 `official-cas-reuse-references.json` 中；`bat doctor cas` 可只读诊断既有 CAS 目录、对象数、对象字节数和元数据库文件状态。
|
||||
- 新 release 发布后会生成 `official-resource-changes.json`、`official-parse-cache.json`、`official-textunit-index.json`、`official-textunit-tasks.json`、`crowdin-translation-handoff.json`、`crowdin-textunit-queue.json`、`translation-tasks.sqlite` 和 `translation-handoff.json`；其中 TextUnit/Crowdin 队列只使用 Added/Modified 资源，不调用 Crowdin 网络 API，离线 TextUnit 翻译任务可通过 `translation.tasks` / `translation.handoff` RPC 或 CLI 查询状态、跳过/失败原因和 provider run 交接。
|
||||
- `translation.worker.run` 已提供 Rust `bat` 的 mock/Crowdin provider worker，支持 lease、失败重试和 TextUnit 译文结果落库；尚缺 Translation Memory、Glossary 和完整 Provider 扩展体系。
|
||||
- `translation.worker.run` 已提供 Rust `bat` 的 mock/Crowdin provider worker，支持 lease、失败重试、TextUnit 译文结果落库和 Translation Memory V1；TM 独立于 release task 库，支持 candidate/trusted、完整 context exact match、显式 confirm 和 provenance 查询。Glossary、模糊匹配和完整 Provider 扩展体系仍待实现。
|
||||
- `LocalizedPatchService` 已具备受支持的 UnityFS localized patch 发布/回滚能力：在 `--localized-output` / `BAT_LOCALIZED_OUTPUT` 配置的独立汉化目录 staging 中复制官方 release、应用 TextAsset、TypeTree string field 或 managed-reference string field patch、写入带 TextUnit/provider/review/rollback trace 的 `localized-patch-manifest.json`，校验后发布到 `versions/<id>` 并切换 `current`，也可显式 rollback。
|
||||
- `bat-patch` 已具备通用 Patch 基础：确定性 Binary hunk diff/apply、RFC 6902 JSON Patch apply、UTF-8 Text Patch、Patch manifest、BLAKE3/size 完整性校验和 rollback 元数据；文件级 `patch.apply` RPC / `patch-apply` CLI 与 UnityFS TextAsset / TypeTree string / TypeTree 语义字段写入入口已开放，TypeTree 语义字段支持基础标量、固定 Unity float/int/hash 值类型的 leaf/direct-child 形态、PPtr、managed-reference registry payload 字符串、object 字段组合、unknown fixed-size raw bytes 同长度替换和 TypeTree schema 支撑的 array/vector/map 整体替换；TextUnit 提取会把 managed-reference 类型信息保留为上下文而非翻译文本，受支持 localized 发布通过独立 manifest/staging/current 流程完成。
|
||||
- `bat-ffi` 可选无状态 C ABI 兼容层：仅保留 Manifest inspect 和官方 sync plan 的粗粒度 JSON helper，不作为 Go CLI 或生产同步的主集成边界。
|
||||
@@ -31,7 +31,7 @@
|
||||
- `bat-api` 完整游戏业务 API / launcher 安装包更新全链仍未完成；资源 CDN、HTTP 控制面、launcher 资源引导兼容和内嵌 dashboard MVP 已可用。
|
||||
- 完整 AssetBundle 对象级解析（UnityFS 解包、object table、TypeTree node 元数据、TextAsset bytes、MonoBehaviour/ScriptableObject 基础 TypeTree 字段级解析已起步；复杂字段覆盖、发布级重打包和 Patch 发布统一仍未完成）。
|
||||
- 复杂 AssetBundle 重打包和完整翻译资产编排仍未完成；当前通用 Binary/JSON/Text Patch 基础已在 crate 层可用，localized 发布仅开放已验证 TextUnit 对应的 UnityFS 文本字段。
|
||||
- Translation Memory、Glossary 和完整 Provider 扩展体系仍未实现。
|
||||
- Translation Memory、Glossary 和完整 Provider 扩展体系：其中 Translation Memory V1 已由 Rust `bat` 持有；仍未实现的是 Glossary、模糊匹配、完整 Provider 扩展体系和完整 Web 协作后台。
|
||||
- SDK、完整 Web 协作后台。
|
||||
|
||||
详细状态见：
|
||||
@@ -107,7 +107,7 @@ cargo run -p bat-infrastructure --bin bat -- reload
|
||||
cargo run -p bat-infrastructure --bin bat -- stop
|
||||
```
|
||||
|
||||
`status`、`stop`、`restart`、`logs`、`reload`、默认形态的 `refresh` 和默认形态的 `repair` 会优先连接 live RPC socket；socket 不可用时，状态和停止命令会回退到 PID/状态文件兼容路径。`restart` 会通过 Rust lifecycle controller 复用 CLI restart 路径替换后台进程；`reload` 不再强制重启进程，而是让后台 watch 循环重新自动发现并执行强制刷新：空闲睡眠时立即唤醒，正在同步时排队到当前轮结束后执行。确实需要替换启动参数时使用 `restart` 或给 `reload` 显式传入同步参数。后台 daemon 正在管理某个资源目录时，前台 `run/watch/refresh/repair` 不能直接写同一目录；默认形态的 `refresh`/`repair` 会改走 RPC，显式参数导致无法走 RPC 时需要先 `stop`。
|
||||
`status`、`stop`、`restart`、`logs`、`reload`、默认形态的 `refresh` 和默认形态的 `repair` 会优先连接 live RPC socket；socket 不可用时，状态和停止命令会回退到 PID/状态文件兼容路径。`restart` 会通过 Rust lifecycle controller 复用 CLI restart 路径替换后台进程；`reload` 不再强制重启进程，而是让后台 watch 循环重新自动发现并执行强制刷新：空闲睡眠时立即唤醒，正在同步时排队到当前轮结束后执行。确实需要替换启动参数时使用 `restart`，或给 `reload` 显式传入同步、输出、worker/TM 等 daemon 启动参数。后台 daemon 正在管理某个资源目录时，前台 `run/watch/refresh/repair` 不能直接写同一目录；默认形态的 `refresh`/`repair` 会改走 RPC，显式参数导致无法走 RPC 时需要先 `stop`。
|
||||
|
||||
`bat` 会拒绝危险输出目录、路径逃逸和现有 symlink 路径组件；下载目标、`.part`、manifest、snapshot、PID、status、log 和控制锁文件不会跟随 symlink，daemon 状态类文件默认以 `0600` 权限创建。
|
||||
|
||||
@@ -141,7 +141,7 @@ make official-smoke
|
||||
|
||||
该 smoke 会执行 dry-run plan、首次全量拉取、二次 `up_to_date` 检查、本地文件破坏后的 `repair`、repair 后 `verify`，并在 `report/SMOKE_REPORT.md` 记录命令、输出目录、active release、文件数量、release 大小和被破坏文件。大型官方资源文件不纳入 Git。
|
||||
|
||||
生产官方资源输出目录和汉化产物目录都必须使用独立目录，不要指向已有客户端目录，也不要指向 `/home/wanye/D/BlueArchive` 这类人工维护或开发资源目录。需要覆盖官方原版资源位置时，用 `--output <资源目录>` 或 `.env` 中的 `BAT_OUTPUT`；需要覆盖汉化产物位置时，用 `--localized-output <目录>` 或 `.env` 中的 `BAT_LOCALIZED_OUTPUT`；需要启用官方 release 导入 CAS/索引时，用 `--import-repository`，并可用 `--import-cas-root`、`--import-resource-db` 或 `.env` 中的 `BAT_IMPORT_CAS_ROOT`、`BAT_IMPORT_RESOURCE_DB` 覆盖默认路径；需要覆盖后台状态目录时，用 `--state-dir <状态目录>`。
|
||||
生产官方资源输出目录和汉化产物目录都必须使用独立目录，不要指向已有客户端目录，也不要指向 `/home/wanye/D/BlueArchive` 这类人工维护或开发资源目录。需要覆盖官方原版资源位置时，用 `--output <资源目录>` 或 `config.toml` 中 `[resource].output_root` / 环境变量 `BAT_OUTPUT`；需要覆盖汉化产物位置时，用 `--localized-output <目录>` 或 `config.toml` 中 `[localized].output_root` / 环境变量 `BAT_LOCALIZED_OUTPUT`；需要启用官方 release 导入 CAS/索引时，用 `--import-repository`，并可用 `config.toml` 中 `[repository].import_cas_root`、`[repository].import_resource_repository_path` 或环境变量 `BAT_IMPORT_CAS_ROOT`、`BAT_IMPORT_RESOURCE_DB` 覆盖默认路径；需要覆盖后台状态目录时，用 `--state-dir <状态目录>` 或 `config.toml` 中 `[runtime].state_dir`。
|
||||
|
||||
---
|
||||
|
||||
@@ -191,10 +191,10 @@ BlueArchiveToolkit/
|
||||
1. 维护并联调 Go `bat-api` 资源 bootstrap/分发入口和内嵌 dashboard；`cmd/bat` 仅有 `doctor`、`manifest inspect`、`sync plan` 试验能力，不应误写成完整产品 CLI。
|
||||
2. 补齐 AssetBundle UnityFS 引擎级解析。
|
||||
3. 扩展 Addressables catalog 解析覆盖，继续用真实形态 fixture/golden 锁定行为。
|
||||
4. 基于 `translation.worker.run` provider worker 推进翻译记忆、完整 Patch 构建和发布/回滚闭环。
|
||||
4. 基于 `translation.worker.run` provider worker 扩展 Glossary、完整 Patch 构建和发布/回滚闭环。
|
||||
5. 按 smoke runbook 在具备网络和磁盘窗口的环境中执行真实官方全量拉取，并保留本地报告。
|
||||
|
||||
当前已提供直接调用 bat-api 鉴权接口的内嵌 dashboard；完整 Web 协作后台仍应在翻译记忆、权限模型和持久化 API 明确后推进。
|
||||
当前已提供直接调用 bat-api 鉴权接口的内嵌 dashboard；完整 Web 协作后台仍应在 TM 扩展、权限模型和持久化 API 明确后推进。
|
||||
|
||||
---
|
||||
|
||||
|
||||
+35
-7
@@ -73,6 +73,7 @@ HTTPS_PROXY=http://user:pass@127.0.0.1:7890 bat --auto-discover --daemon
|
||||
| `clean-stable` | 清理 `.part`/`.tmp`/失效锁、PID、socket（daemon 运行中会拒绝执行） |
|
||||
|
||||
`status`/`stop`/`logs`/`reload` 和默认形态的 `refresh` 优先走 `bat.sock` JSON-RPC；socket 不可用时 `status`/`stop` 回退到 PID/状态文件兼容路径。
|
||||
需要替换 daemon 启动参数时使用 `restart`，或给 `reload` 显式传入同步、输出、worker/TM 等启动参数；未显式传参的 `restart` 复用上次保存的启动命令。
|
||||
|
||||
Rust `bat` 工作流的完整命令、工作台字段、重打包 spec、调度计划和
|
||||
`bat-api` 调度接口见 [`docs/guides/bat-workflows.md`](docs/guides/bat-workflows.md)。
|
||||
@@ -149,6 +150,8 @@ BAT_API_SKIP_ENV_FILE=1 go run ./cmd/bat-api \
|
||||
| `GET /admin/parse/errors?...` | 分页查询当前 release 解析错误；需要管理 token |
|
||||
| `GET /admin/translation/tasks?limit=100&worker_status=failed` | 读取/过滤 Rust 翻译任务和 provider worker 状态；需要管理 token |
|
||||
| `GET /admin/translation/handoff` | 读取当前 release 的完整翻译交接视图；需要管理 token |
|
||||
| `GET /admin/translation/memory/summary` | 读取 Rust TM schema、记录总数及 candidate/trusted 等状态计数；需要管理 token |
|
||||
| `GET /admin/translation/memory/query?source_text=...&source_context=...&limit=100` | 按 raw source/context 查询 Rust TM 记录、复用判定和 provenance；需要管理 token |
|
||||
| `GET /admin/translation/status` | 读取当前汉化 release、current 指针和 workflow 状态；需要管理 token |
|
||||
| `POST /admin/control/{action}` | 经白名单转发 Rust `bat` 控制请求；见下文 |
|
||||
|
||||
@@ -183,6 +186,7 @@ launcher 兼容端点只服务启动前资源发现。它们复用 Rust `bat` sn
|
||||
| `translation-task-update` | `translation.task.update` | `{ "task_id": "...", "status": "completed", "provider": "manual", "provider_run_id": "...", "translation_results": [{ "unit_id": "...", "source_text": "...", "translated_text": "..." }] }` | `202` + 当前任务记录 |
|
||||
| `translation-worker-run` | `translation.worker.run` | `{ "provider": "mock", "concurrency": 8, "max_tasks": 2 }` | `202` + worker task |
|
||||
| `translation-proofread` | `translation.proofread` | 无 | `202` + 汉化状态 |
|
||||
| `translation-memory-confirm` | `translation.memory.confirm` | `{ "record_id": "...", "reviewer": "...", "reason": "..." }` | `202` + 已确认的 TM 记录 |
|
||||
| `localized-publish` | `localized.publish` | `{ "translation_file": "...", "localized_release_id": "..." }` 或 `{ "from_worker": true, "localized_release_id": "..." }` | `202` + localized release manifest |
|
||||
| `localized-rollback` | `localized.rollback` | 可选 `{ "localized_release_id": "..." }` | `202` + rollback report |
|
||||
|
||||
@@ -272,16 +276,40 @@ curl -i -H 'Range: bytes=0-1023' \
|
||||
- 强制刷新：每天北京时间（UTC+8）`03:00`、`16:00`、`18:00` 各一次。
|
||||
- 状态类文件默认 `0600` 权限，读写不跟随 symlink。
|
||||
|
||||
### 配置文件（`.env`，无参启动）
|
||||
### 配置文件（config.toml，无参启动）
|
||||
|
||||
`bat` 首次启动时会在**二进制所在目录**释放一个 `.env` 配置模板（`0600` 权限，已存在则不动）。之后每次启动自动加载该文件，把其中的键作为进程环境变量（不覆盖已存在的环境变量），因此编辑 `.env` 后直接运行 `bat`（无参数）即可按配置启动。
|
||||
`bat` 首次启动时会在**二进制所在目录**释放一个 `config.toml.example` 配置模板（`0600` 权限，已存在则不动）。程序只读取同目录下的 `config.toml`；`config.toml.example` 只是模板，不会被自动读取，也不会自动复制或重命名为 `config.toml`。没有 `config.toml` 时，程序继续使用进程环境变量和内置默认值启动。
|
||||
|
||||
- 优先级：**命令行参数 > 进程环境变量 > `.env` > 内置默认值**。
|
||||
- 语法：每行 `KEY=VALUE`；`#` 开头为注释；值两侧成对引号会剥除；空值视为未设置。
|
||||
- 支持的键：`BAT_OUTPUT`、`BAT_STATE_DIR`、`BAT_AUTO_DISCOVER`、`BAT_WATCH`、`BAT_DAEMON`、`BAT_PROXY`、`BAT_NO_PROXY`、`BAT_INTERVAL_SECONDS`、`BAT_ERROR_RETRY_SECONDS`、`BAT_APP_VERSION`、`BAT_CONNECTION_GROUP`、`BAT_LAUNCHER_VERSION`、`BAT_PLATFORMS`、`BAT_CURL`、`BAT_UNZIP`、`BAT_JSON`、`BAT_QUIET_UP_TO_DATE`；也可以直接写 `HTTPS_PROXY` 等通用环境变量（走现有代理自动检测）。布尔值支持 `1/0/true/false/yes/no/on/off`。
|
||||
- `BAT_WATCH` / `BAT_DAEMON` 只对无子命令的 `bat` 生效（两者同时为 `1` 时 daemon 优先）；命令行显式传入 `--watch` / `--daemon` / `--dry-run` 时 `.env` 的模式开关让位。`status` / `verify` 等子命令不受它们影响。
|
||||
由于 `config.toml` 可能包含代理凭据，Unix 下实际 `config.toml` 必须保持 `0600` 或更严格；权限过宽时程序会拒绝读取。
|
||||
|
||||
- 优先级：**命令行参数 > 进程环境变量 > `config.toml` > 内置默认值**。
|
||||
- `config.toml` 的字段按职责分组：`[runtime]`、`[resource]`、`[localized]`、`[repository]`、`[network]`、`[translation.worker]`。
|
||||
- 现有 `BAT_*` 环境变量仍然有效，可继续覆盖 `config.toml` 中的同名配置。
|
||||
- `BAT_SKIP_ENV_FILE` 已废弃且不再影响启动。
|
||||
- 支持的环境变量：`BAT_OUTPUT`、`BAT_LOCALIZED_OUTPUT`、`BAT_STATE_DIR`、`BAT_AUTO_DISCOVER`、`BAT_WATCH`、`BAT_DAEMON`、`BAT_IMPORT_REPOSITORY`、`BAT_IMPORT_CAS_ROOT`、`BAT_IMPORT_RESOURCE_DB`、`BAT_PROXY`、`BAT_NO_PROXY`、`BAT_INTERVAL_SECONDS`、`BAT_ERROR_RETRY_SECONDS`、`BAT_APP_VERSION`、`BAT_CONNECTION_GROUP`、`BAT_LAUNCHER_VERSION`、`BAT_PLATFORMS`、`BAT_CURL`、`BAT_DOWNLOAD_CONCURRENCY`、`BAT_UNZIP`、`BAT_JSON`、`BAT_QUIET_UP_TO_DATE`、`BAT_TRANSLATION_PROVIDER`、`BAT_TRANSLATION_FIXTURE`、`BAT_TRANSLATION_MEMORY_PATH`、`BAT_TRANSLATION_CONCURRENCY`、`BAT_TRANSLATION_MAX_ATTEMPTS`、`BAT_TRANSLATION_LEASE_SECONDS`、`BAT_TRANSLATION_RETRY_BACKOFF_SECONDS`、`BAT_TRANSLATION_MAX_TASKS`、`BAT_TRANSLATION_WORKER_ID`；也可以直接写 `HTTPS_PROXY` 等通用环境变量（走现有代理自动检测）。布尔值支持 `1/0/true/false/yes/no/on/off`。
|
||||
- `BAT_WATCH` / `BAT_DAEMON` 只对无子命令的 `bat` 生效（两者同时为 `1` 时 daemon 优先）；命令行显式传入 `--watch` / `--daemon` / `--dry-run` 时运行模式设置让位。`status` / `verify` 等子命令不受它们影响。
|
||||
- 已运行的 daemon 不会热读 `config.toml`；默认 `reload` 只唤醒后台重新发现和刷新。需要应用配置文件变更时，使用带显式启动参数的 `restart`/`reload`，或先 `stop` 再重新启动 daemon。
|
||||
- `BAT_REDIS_URL` / `BAT_REDIS_PASSWORD` 为**预留键**：Redis 任务后端尚未接入，当前任务历史持久化在 `<state-dir>/bat-tasks.json`。
|
||||
- 设 `BAT_SKIP_ENV_FILE=1` 可让 `bat` 完全跳过 `.env` 的生成与加载。
|
||||
|
||||
### Translation Memory V1
|
||||
|
||||
Translation Memory 由 Rust `bat` 独立持有，默认路径为
|
||||
`<output>/translation-memory.sqlite`，不在 `versions/<id>` 内，也不使用当前 release
|
||||
的 `translation-tasks.sqlite`。可通过 `[translation.worker].translation_memory_path`、
|
||||
`BAT_TRANSLATION_MEMORY_PATH` 或 `--translation-memory-path` 覆盖。
|
||||
|
||||
```bash
|
||||
bat i18n memory summary
|
||||
bat i18n memory query --tm-source-text '原始文本' --tm-context-json '{"destination":"Table.bytes","archive_entry":"","field_path":"Text"}'
|
||||
bat i18n memory confirm --tm-record-id 'tm-...' --tm-reviewer 'operator' --tm-reason '人工校对通过'
|
||||
```
|
||||
|
||||
只有 raw source 完全相同、完整 context 完全相同且状态为 `trusted` 的记录会被 worker
|
||||
自动复用。provider 输出写入先是 `candidate`；manual task result 即使 completed 也不会自动
|
||||
建立 TM 或 trusted。查询、诊断和显式 confirm 对应 Rust
|
||||
RPC `translation.memory.summary`、`translation.memory.query`、`translation.memory.confirm`。
|
||||
context 不完整或不一致、normalized source 辅助命中和 workflow `proofread` 都不会自动
|
||||
建立 trusted 记录。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -5,5 +5,6 @@ HTTP surface. The running service also exposes the same contract at
|
||||
`GET /openapi.yaml`.
|
||||
|
||||
This contract covers resource bootstrap, launcher resource compatibility,
|
||||
server-info rewrite, CDN-shaped resource bytes, auth schemes, and the reserved
|
||||
admin panel entry. It does not describe a full game business API.
|
||||
server-info rewrite, CDN-shaped resource bytes, auth schemes, and the admin
|
||||
panel's Rust-forwarded translation/TM management routes. It does not describe
|
||||
a full game business API.
|
||||
|
||||
@@ -404,6 +404,56 @@ paths:
|
||||
description: Missing or invalid admin token.
|
||||
"503":
|
||||
description: Rust bat translation backend is unavailable.
|
||||
/admin/translation/memory/summary:
|
||||
get:
|
||||
summary: Read Rust-owned Translation Memory summary
|
||||
parameters:
|
||||
- name: translation_memory_path
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Translation Memory availability and candidate/trusted counts.
|
||||
"401":
|
||||
description: Missing or invalid admin token.
|
||||
"503":
|
||||
description: Rust bat Translation Memory backend is unavailable.
|
||||
/admin/translation/memory/query:
|
||||
get:
|
||||
summary: Query Rust-owned Translation Memory records
|
||||
parameters:
|
||||
- name: source_text
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: source_context
|
||||
in: query
|
||||
description: JSON object whose values are strings.
|
||||
schema:
|
||||
type: string
|
||||
- name: limit
|
||||
in: query
|
||||
schema:
|
||||
type: integer
|
||||
format: int64
|
||||
minimum: 1
|
||||
maximum: 1000
|
||||
default: 100
|
||||
- name: translation_memory_path
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Translation Memory matches with reuse decision and provenance.
|
||||
"400":
|
||||
description: Missing source text or invalid context/limit.
|
||||
"401":
|
||||
description: Missing or invalid admin token.
|
||||
"503":
|
||||
description: Rust bat Translation Memory backend is unavailable.
|
||||
/admin/translation/status:
|
||||
get:
|
||||
summary: Read Rust-owned localized release status
|
||||
@@ -423,7 +473,7 @@ paths:
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh, schedule-add, schedule-update, schedule-remove, schedule-run, task-cancel, translation-task-update, translation-worker-run, translation-proofread, localized-publish, localized-rollback]
|
||||
enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh, schedule-add, schedule-update, schedule-remove, schedule-run, task-cancel, translation-task-update, translation-worker-run, translation-proofread, translation-memory-confirm, localized-publish, localized-rollback]
|
||||
requestBody:
|
||||
required: false
|
||||
content:
|
||||
@@ -514,6 +564,14 @@ paths:
|
||||
minimum: 1
|
||||
worker_id:
|
||||
type: string
|
||||
translation_memory_path:
|
||||
type: string
|
||||
record_id:
|
||||
type: string
|
||||
reviewer:
|
||||
type: string
|
||||
reason:
|
||||
type: string
|
||||
translation_file:
|
||||
type: string
|
||||
from_worker:
|
||||
|
||||
@@ -4,6 +4,7 @@ pub mod game_client;
|
||||
pub mod game_version;
|
||||
pub mod resource;
|
||||
pub mod translation;
|
||||
pub mod translation_memory;
|
||||
|
||||
pub use game_client::{ClientStatus, GameClient, GameRegion};
|
||||
pub use game_version::{GameVersion, UnityVersion};
|
||||
@@ -14,3 +15,8 @@ pub use translation::{
|
||||
ExtractedText, SourceText, TextContext, TextMetadata, TextSource, TranslatedText,
|
||||
TranslationStatus,
|
||||
};
|
||||
pub use translation_memory::{
|
||||
TranslationMemoryContext, TranslationMemoryDraft, TranslationMemoryEntry,
|
||||
TranslationMemoryMatch, TranslationMemoryMatchKind, TranslationMemorySourceKind,
|
||||
TranslationMemorySourceTrace, TranslationMemorySummary, TranslationMemoryTrustStatus,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
//! Translation Memory 领域对象。
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
/// TM 记录的来源类型。
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum TranslationMemorySourceKind {
|
||||
/// 来自 provider 输出。
|
||||
Provider,
|
||||
/// 来自人工确认。
|
||||
Manual,
|
||||
/// 来自外部导入。
|
||||
Imported,
|
||||
}
|
||||
|
||||
impl TranslationMemorySourceKind {
|
||||
/// 返回稳定的持久化标签。
|
||||
pub const fn as_str(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Provider => "provider",
|
||||
Self::Manual => "manual",
|
||||
Self::Imported => "imported",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// TM 记录的可信状态。
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum TranslationMemoryTrustStatus {
|
||||
/// 候选记录，不能自动复用。
|
||||
Candidate,
|
||||
/// 已确认可信，可在强匹配时自动复用。
|
||||
Trusted,
|
||||
/// 已被后续记录取代。
|
||||
Superseded,
|
||||
/// 已明确拒绝。
|
||||
Rejected,
|
||||
}
|
||||
|
||||
impl TranslationMemoryTrustStatus {
|
||||
/// 返回稳定的持久化标签。
|
||||
pub const fn as_str(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Candidate => "candidate",
|
||||
Self::Trusted => "trusted",
|
||||
Self::Superseded => "superseded",
|
||||
Self::Rejected => "rejected",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// TM 查询结果的匹配类型。
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum TranslationMemoryMatchKind {
|
||||
/// 原始 source 和上下文都完全匹配，且记录可信，可自动复用。
|
||||
StrongExact,
|
||||
/// 原始 source 完全匹配，但上下文不同或不足，不能自动复用。
|
||||
CandidateExact,
|
||||
/// 原始 source 匹配，但上下文不兼容，不能自动复用。
|
||||
SourceOnly,
|
||||
}
|
||||
|
||||
impl TranslationMemoryMatchKind {
|
||||
/// 返回稳定的查询结果标签。
|
||||
pub const fn as_str(&self) -> &'static str {
|
||||
match self {
|
||||
Self::StrongExact => "strong_exact",
|
||||
Self::CandidateExact => "candidate_exact",
|
||||
Self::SourceOnly => "source_only",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 稳定的上下文键值。
|
||||
pub type TranslationMemoryContext = BTreeMap<String, String>;
|
||||
|
||||
/// TM 记录的 TextUnit / provider 溯源信息。
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct TranslationMemorySourceTrace {
|
||||
/// 源官方 release ID。
|
||||
pub official_release_id: String,
|
||||
/// 来源 TextUnit ID。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub unit_id: Option<String>,
|
||||
/// 来源任务 ID。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub task_id: Option<String>,
|
||||
/// 源资源 destination。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub destination: Option<String>,
|
||||
/// 源 ZIP/archive entry。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub archive_entry: Option<String>,
|
||||
/// Unity serialized file。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub serialized_file: Option<String>,
|
||||
/// Unity object path ID。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub path_id: Option<i64>,
|
||||
/// Unity class ID。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub class_id: Option<i32>,
|
||||
/// TypeTree 字段路径。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub field_path: Option<String>,
|
||||
/// TextUnit format。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub format: Option<String>,
|
||||
/// TextAsset 名称。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub asset_name: Option<String>,
|
||||
/// TextUnit 来源类型。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub text_source_kind: Option<String>,
|
||||
/// 源 URL。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub source_url: Option<String>,
|
||||
}
|
||||
|
||||
/// TM 记录的候选输入。
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct TranslationMemoryDraft {
|
||||
/// 原始 source text。
|
||||
pub source_text: String,
|
||||
/// 稳定上下文。
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub source_context: TranslationMemoryContext,
|
||||
/// 译文。
|
||||
pub translated_text: String,
|
||||
/// 译文来源类型。
|
||||
pub translation_source_kind: TranslationMemorySourceKind,
|
||||
/// 源官方 release。
|
||||
pub official_release_id: String,
|
||||
/// 原始 TextUnit / provider 溯源。
|
||||
pub source_trace: TranslationMemorySourceTrace,
|
||||
/// provider。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub provider: Option<String>,
|
||||
/// provider run。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub provider_run_id: Option<String>,
|
||||
/// 创建时间。
|
||||
pub observed_unix_seconds: u64,
|
||||
}
|
||||
|
||||
/// 持久化 TM 记录。
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct TranslationMemoryEntry {
|
||||
/// 稳定记录 ID。
|
||||
pub record_id: String,
|
||||
/// 原始 source text。
|
||||
pub source_text: String,
|
||||
/// source text hash。
|
||||
pub source_hash: String,
|
||||
/// 保守归一化后的 source text，仅用于辅助查询。
|
||||
pub normalized_source_text: String,
|
||||
/// 稳定上下文。
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub source_context: TranslationMemoryContext,
|
||||
/// 上下文 hash。
|
||||
pub source_context_hash: String,
|
||||
/// 译文。
|
||||
pub translated_text: String,
|
||||
/// 译文来源类型。
|
||||
pub translation_source_kind: TranslationMemorySourceKind,
|
||||
/// 当前可信状态。
|
||||
pub trust_status: TranslationMemoryTrustStatus,
|
||||
/// 源官方 release。
|
||||
pub official_release_id: String,
|
||||
/// 原始 TextUnit / provider 溯源。
|
||||
pub source_trace: TranslationMemorySourceTrace,
|
||||
/// provider。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub provider: Option<String>,
|
||||
/// provider run。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub provider_run_id: Option<String>,
|
||||
/// 创建时间。
|
||||
pub created_unix_seconds: u64,
|
||||
/// 更新时间。
|
||||
pub updated_unix_seconds: u64,
|
||||
/// 可信确认时间。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub trusted_unix_seconds: Option<u64>,
|
||||
/// 可信确认人。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub trusted_by: Option<String>,
|
||||
/// 可信确认说明。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub trusted_reason: Option<String>,
|
||||
/// 该记录替代了哪条记录。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub supersedes_record_id: Option<String>,
|
||||
/// 该记录被哪条记录替代。
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub superseded_by_record_id: Option<String>,
|
||||
}
|
||||
|
||||
/// TM 查询结果。
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct TranslationMemoryMatch {
|
||||
/// 记录本体。
|
||||
pub entry: TranslationMemoryEntry,
|
||||
/// 匹配类型。
|
||||
pub match_kind: TranslationMemoryMatchKind,
|
||||
/// 是否允许自动复用。
|
||||
pub can_auto_reuse: bool,
|
||||
}
|
||||
|
||||
/// TM 仓储摘要。
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct TranslationMemorySummary {
|
||||
/// schema 版本。
|
||||
pub schema_version: u32,
|
||||
/// 记录总数。
|
||||
pub record_count: u64,
|
||||
/// 可信记录数。
|
||||
pub trusted_count: u64,
|
||||
/// 候选记录数。
|
||||
pub candidate_count: u64,
|
||||
/// 已替代记录数。
|
||||
pub superseded_count: u64,
|
||||
/// 已拒绝记录数。
|
||||
pub rejected_count: u64,
|
||||
}
|
||||
@@ -4,8 +4,10 @@
|
||||
|
||||
pub mod cas_repository;
|
||||
pub mod resource_repository;
|
||||
pub mod translation_memory_repository;
|
||||
pub mod translation_repository;
|
||||
|
||||
pub use cas_repository::CasRepository;
|
||||
pub use resource_repository::ResourceRepository;
|
||||
pub use translation_memory_repository::TranslationMemoryRepository;
|
||||
pub use translation_repository::TranslationRepository;
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
//! Translation Memory 仓储契约。
|
||||
|
||||
use crate::domain::{
|
||||
TranslationMemoryContext, TranslationMemoryDraft, TranslationMemoryEntry,
|
||||
TranslationMemoryMatch, TranslationMemorySummary,
|
||||
};
|
||||
use async_trait::async_trait;
|
||||
|
||||
/// 跨 official release 持久化的 Translation Memory 仓储。
|
||||
///
|
||||
/// 该契约只描述 V1 的精确查询和明确人工确认。仓储实现不得把
|
||||
/// `TranslationTaskStatus::Completed` 或 provider 成功隐式解释为 trusted。
|
||||
#[async_trait]
|
||||
pub trait TranslationMemoryRepository: Send + Sync {
|
||||
/// 保存一条 provider/manual/imported 译文候选。
|
||||
///
|
||||
/// 相同 source、上下文、译文、来源 release 和来源类型的重复写入必须幂等；
|
||||
/// 已 trusted 的记录不得被普通候选静默覆盖。
|
||||
async fn upsert_candidate(
|
||||
&self,
|
||||
draft: TranslationMemoryDraft,
|
||||
) -> crate::Result<TranslationMemoryEntry>;
|
||||
|
||||
/// 按原始 source text 和上下文查询精确匹配。
|
||||
///
|
||||
/// 实现可以返回 source 归一化后但原文不同的辅助候选，但这类结果不能自动复用。
|
||||
async fn find_matches(
|
||||
&self,
|
||||
source_text: &str,
|
||||
source_context: &TranslationMemoryContext,
|
||||
limit: usize,
|
||||
) -> crate::Result<Vec<TranslationMemoryMatch>>;
|
||||
|
||||
/// 显式确认一条记录为 trusted。
|
||||
async fn confirm(
|
||||
&self,
|
||||
record_id: &str,
|
||||
reviewer: &str,
|
||||
reason: Option<String>,
|
||||
) -> crate::Result<TranslationMemoryEntry>;
|
||||
|
||||
/// 按稳定记录 ID 读取一条 TM 记录。
|
||||
async fn find(&self, record_id: &str) -> crate::Result<TranslationMemoryEntry>;
|
||||
|
||||
/// 读取数据库和记录统计。
|
||||
async fn summary(&self) -> crate::Result<TranslationMemorySummary>;
|
||||
}
|
||||
+4
-1
@@ -18,7 +18,10 @@ Go `cmd/bat-api` 是资源 bootstrap、已发布资源分发和鉴权控制服
|
||||
- `/v1/bootstrap`、`/v1/launcher/bootstrap`、`/v1/release`、`/v1/resources`
|
||||
- `/v1/server-info` 和 CDN 形状资源路径
|
||||
- `/api/launcher/game/config` 兼容端点
|
||||
- `/admin/` 与白名单 `/admin/control/{action}`
|
||||
- `/admin/` 与白名单 `/admin/control/{action}`；其中翻译管理面包含
|
||||
`/admin/translation/tasks`、`/admin/translation/handoff`、
|
||||
`/admin/translation/memory/summary`、`/admin/translation/memory/query` 和
|
||||
`translation-memory-confirm` 转发
|
||||
- `/openapi.yaml`
|
||||
|
||||
HTTP 路由的 OpenAPI 文本由 `internal/api/openapi.go` 提供，运行中的服务也可
|
||||
|
||||
@@ -152,7 +152,8 @@ current symlink → official-sync-snapshot.json + official-download-manifest.jso
|
||||
**Go 当前职责**：
|
||||
|
||||
- `bat-api` 通过 `bat.sock` RPC 读取 Rust 已发布 release、manifest、snapshot 和状态。
|
||||
- 提供资源 bootstrap、server-info 改写、只读 CDN path、readiness、OpenAPI 和白名单管理转发。
|
||||
- 提供资源 bootstrap、server-info 改写、只读 CDN path、readiness、OpenAPI 和白名单管理转发；
|
||||
翻译任务与 TM 管理接口只通过 Rust RPC 代理，不在 Go 侧持有状态。
|
||||
- 不运行另一套同步器，不直接管理官方下载、staging、version-state、CAS 或解析状态。
|
||||
|
||||
完整 API、服务编排、Provider 和用户配置属于目标扩展，不能从本节推断为当前已实现。
|
||||
@@ -195,12 +196,13 @@ pub struct ParserRegistry {
|
||||
### 4. 翻译系统（目标扩展，Go；当前 worker 由 Rust `bat` 承担）
|
||||
|
||||
当前已实现的是 Rust `bat` 的离线 TextUnit 队列、mock/Crowdin provider worker、
|
||||
lease/retry 和结果落库；Translation Memory、Glossary 和完整 Provider 体系仍属
|
||||
后续缺口。
|
||||
lease/retry、结果落库和项目级 Translation Memory V1。TM 位于独立 SQLite，按 raw
|
||||
source + 完整 context 做 trusted exact reuse，candidate 必须显式 confirm；Glossary、
|
||||
模糊匹配和完整 Provider 体系仍属后续缺口。
|
||||
|
||||
**架构**：
|
||||
```
|
||||
Text Extractor → Translation Memory (查询) → AI Provider → Glossary (术语替换) → Output
|
||||
Text Extractor → TM exact query → AI Provider → Glossary (后续) → Output
|
||||
↓ ↓
|
||||
PostgreSQL 审核队列
|
||||
```
|
||||
@@ -226,9 +228,10 @@ type TranslationProvider interface {
|
||||
- Azure Translator Provider
|
||||
|
||||
**翻译记忆库**：
|
||||
- 精确匹配：100% 匹配直接使用
|
||||
- 模糊匹配：使用相似度算法（Levenshtein Distance）
|
||||
- 上下文匹配：根据前后文提高匹配准确度
|
||||
- 当前 V1：raw source 完全相同、完整 context 完全相同且记录为 trusted 时自动复用。
|
||||
- provider 输出写入先是 candidate；manual task result 不会自动建立 TM 或 trusted。`bat i18n memory confirm` 显式确认单条记录后才可自动复用。
|
||||
- source、context、release、TextUnit、provider 和 run provenance 保存在 Rust TM SQLite 中。
|
||||
- 模糊匹配、术语优先级和 PostgreSQL 服务化仍不是当前实现。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -26,7 +26,7 @@
|
||||
- 通过 `bat.sock` RPC 发现 Rust 已发布的 `resource_root`、snapshot、manifest
|
||||
和状态；
|
||||
- 提供资源 bootstrap、server-info 改写、只读 CDN path、readiness、OpenAPI
|
||||
以及鉴权后的白名单管理转发；
|
||||
以及鉴权后的白名单管理转发；翻译任务和 TM 管理面只转发 Rust RPC；
|
||||
- 不下载官方资源、不写 staging、不维护 version-state，不复制 CAS、解析器、
|
||||
Patch 核心算法或同步状态机。
|
||||
|
||||
@@ -36,8 +36,8 @@
|
||||
- 新的跨语言控制和查询能力优先增加 Rust RPC contract，再由
|
||||
`internal/backendrpc` 消费。
|
||||
|
||||
4. **完整游戏业务 API、完整 Web 协作后台、Translation Memory、Glossary 和
|
||||
Provider 扩展体系仍是后续目标**，不能从目标架构图推断为当前已实现。
|
||||
4. **完整游戏业务 API、完整 Web 协作后台、Glossary 和 Provider 扩展体系仍是后续目标**；
|
||||
Translation Memory V1 已由 Rust `bat` 持有，不能从目标架构图推断 Go 侧拥有第二份状态。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -208,7 +208,7 @@ catalog 仍按“明确不支持”处理，不把低保真路径伪装成完整
|
||||
|
||||
1. 继续补充 Addressables Windows/Android 真实 catalog 样本和独立二进制格式诊断。
|
||||
2. 继续补充 TypeTree 字段 reader、MonoBehaviour/ScriptableObject 遍历和真实版本差异。
|
||||
3. 基于 `translation.worker.run` 推进翻译记忆和通用 manifest Patch 构建。
|
||||
3. 基于 `translation.worker.run` 扩展 TM/Glossary 和通用 manifest Patch 构建。
|
||||
4. 扩展翻译任务结果在 CAS/ResourceRepository 查询面的索引。
|
||||
5. 在通用 Binary/JSON/Text Patch 基础上继续扩展复杂 AssetBundle 重打包和通用
|
||||
Patch 发布流程统一，保留当前受支持 localized patch 发布/rollback 链路。
|
||||
|
||||
@@ -162,7 +162,7 @@
|
||||
### 3.5 导入到 CAS 和资源仓储
|
||||
|
||||
官方同步下载、校验并发布 release 后，可以通过 `--import-repository` 或
|
||||
`.env` 中 `BAT_IMPORT_REPOSITORY=1` 自动触发 CAS + `ResourceRepository`
|
||||
`config.toml` / 环境变量 `BAT_IMPORT_REPOSITORY=1` 自动触发 CAS + `ResourceRepository`
|
||||
导入：
|
||||
|
||||
1. 读取已发布 release 下的 `official-download-manifest.json`。
|
||||
@@ -180,12 +180,14 @@
|
||||
`available=false`，不会因为查询创建空库。发布后的 TextUnit 队列还会在当前
|
||||
release 根目录写入 `translation-tasks.sqlite`，由版本化 `schema_migrations`
|
||||
管理 queued/running/failed/completed/skipped、provider run、lease、失败分类、
|
||||
重试计划和 TextUnit 级译文结果。`translation.tasks` 优先查询这份状态库，
|
||||
重试计划和 TextUnit 级译文结果。跨 release 的 Translation Memory V1 独立存储在
|
||||
`<output>/translation-memory.sqlite`，记录 raw source/hash、完整 context、candidate/
|
||||
trusted 和 release/TextUnit/provider/run provenance；`translation.tasks` 优先查询这份状态库，
|
||||
`translation.worker.run` 由 Rust worker 回写状态；`translation.task.update` 仍供外部 provider 流程回写状态；
|
||||
没有状态库的旧 release 才回退到 immutable JSON 队列。`bat doctor cas`
|
||||
已提供只读 CAS 根目录、对象目录、元数据库文件和对象统计诊断；`resource.index`
|
||||
已把 release、平台、bundle path 和常用数组 metadata 过滤下推到 SQLite。G-011
|
||||
剩余工作是更丰富的 TextUnit 查询、翻译记忆查询和通用 Patch 发布所需资源视图。
|
||||
剩余工作是更丰富的 TextUnit/TM 查询和通用 Patch 发布所需资源视图。
|
||||
|
||||
对应实现主要在：
|
||||
|
||||
|
||||
@@ -89,8 +89,8 @@ git check-ignore -v Cargo.lock CLAUDE.md AGENTS.md CONTRIBUTING.md
|
||||
当前开发优先推进：
|
||||
|
||||
1. 继续 AssetBundle 复杂对象解析、真实 fixture 和发布级重打包。
|
||||
2. 基于 `translation.worker.run` 推进翻译记忆和通用 manifest Patch 构建。
|
||||
3. 扩展 ResourceRepository 查询面：更丰富的 TextUnit 查询、翻译记忆查询和通用 Patch 发布所需资源视图。
|
||||
2. 基于 `translation.worker.run` 扩展 TM/Glossary 和通用 manifest Patch 构建。
|
||||
3. 扩展 ResourceRepository 查询面：更丰富的 TextUnit/TM 查询和通用 Patch 发布所需资源视图。
|
||||
4. 按 `docs/guides/official-full-pull-smoke.md` 在隔离目录执行真实官方网络全量下载 smoke，并保留运行报告。
|
||||
5. 在资源和翻译契约稳定后推进完整 Web 协作后台和完整游戏业务 API。
|
||||
|
||||
|
||||
@@ -8,8 +8,9 @@ BlueArchive Toolkit 的部署文档分为当前可用模式和目标模式：
|
||||
本地资源状态使用文件和 SQLite。
|
||||
2. **官方资源同步生产任务**：当前可用，运行 Rust `bat --watch` 或 RPC/daemon 模式。
|
||||
3. **bat-api 资源 bootstrap / 分发服务**：当前可用，和 Rust `bat` 在同一服务器/容器环境运行，经 `bat.sock` RPC 获取当前 `resource_root`。
|
||||
4. **可选数据库开发环境**：PostgreSQL/Redis 只服务于未来的 Go 服务层、Translation
|
||||
Memory、Glossary 和完整 Provider 扩展，不是当前 `bat` / `bat-api` 的生产运行依赖。
|
||||
4. **可选数据库开发环境**：PostgreSQL/Redis 只服务于未来的 Go 服务层、Glossary 和完整
|
||||
Provider 扩展，不是当前 `bat` / `bat-api` 的生产运行依赖；当前 Translation Memory V1
|
||||
使用 `<output>/translation-memory.sqlite`。
|
||||
5. **完整单机/分布式部署**：尚未提供。完整游戏业务 API、数据库迁移和 Web 未实现前，不把它作为可执行部署方案。
|
||||
|
||||
---
|
||||
@@ -38,7 +39,7 @@ docker compose -f deployments/docker-compose.dev.yml --profile local-db up -d
|
||||
## 模式 2：可选数据库开发环境（目标能力）
|
||||
|
||||
PostgreSQL 和 Redis 不是当前 `bat` / `bat-api` 的生产运行依赖。本模式只用于未来
|
||||
服务层、Translation Memory、Glossary 或 Provider 扩展的开发验证，不能作为当前
|
||||
服务层、Glossary 或 Provider 扩展的开发验证，不能作为当前
|
||||
资源同步或资源分发的部署前置条件。
|
||||
|
||||
### 远程开发连接
|
||||
|
||||
@@ -257,7 +257,7 @@ cargo run -p bat-infrastructure --bin bat -- \
|
||||
--import-resource-db /tmp/bat-test-resources.sqlite
|
||||
```
|
||||
|
||||
对应 `.env` / 环境变量键为 `BAT_IMPORT_REPOSITORY`、
|
||||
对应 `config.toml` / 环境变量键为 `BAT_IMPORT_REPOSITORY`、
|
||||
`BAT_IMPORT_CAS_ROOT` 和 `BAT_IMPORT_RESOURCE_DB`。只读查询命令：
|
||||
|
||||
```bash
|
||||
|
||||
@@ -218,7 +218,7 @@ cargo run -p bat-infrastructure --example official_pull_plan -- \
|
||||
- 非 dry-run 且启用 `--auto-discover` 时，成功发布的 release 会包含 `official-launcher-bootstrap.json`；up-to-date 轮询发现当前 release 缺少该文件时会补写。官方 launcher/server-info 已更新但 client-patch 资源尚未开放时，不切换 `current`，只在输出根写入 `official-launcher-bootstrap.pending.json` 作为维护期证据。
|
||||
- 校验和发布完成后会先对比上一完整 release 与当前 release 的 `official-download-manifest.json`，写出 `<output>/current/official-resource-changes.json` 和 `<output>/current/crowdin-translation-handoff.json`。同一 destination 只有 size 或 BLAKE3 改变才算 modified；仅 URL/CDN 根变化但内容一致不会触发解析/翻译候选。新增+变更资源进入解析和 Crowdin 翻译 handoff，删除资源只进入差异记录；当前不会直接调用 Crowdin API。
|
||||
- 随后会刷新 `<output>/current/official-parse-cache.json`。解析缓存从 `official-download-manifest.json` 的全部条目出发，处理直接 UnityFS bundle 和 zip 内 UnityFS 条目；catalog、hash、媒体等非 UnityFS 文件记录为不支持，不视为同步失败。新 release 会刷新解析缓存；远端和本地都 up-to-date 且已有有效解析缓存时只读取摘要，不重复解析。
|
||||
- 需要将已校验官方 release 导入 CAS + SQLite ResourceRepository 时，使用 `--import-repository` 或 `.env` 中 `BAT_IMPORT_REPOSITORY=1`；默认 CAS 为 `<output>/.cas`，默认索引为 `<output>/resources.sqlite`，可用 `--import-cas-root` / `BAT_IMPORT_CAS_ROOT` 和 `--import-resource-db` / `BAT_IMPORT_RESOURCE_DB` 覆盖。`resource.index` RPC 可查询现有索引，索引不存在时返回 `available=false`，不会创建空库；`bat doctor cas --output <output>` 或 `bat doctor cas --import-cas-root <path>` 可只读检查既有 CAS 根目录、对象目录、元数据库文件和对象统计。
|
||||
- 需要将已校验官方 release 导入 CAS + SQLite ResourceRepository 时，使用 `--import-repository` 或 `config.toml` / 环境变量 `BAT_IMPORT_REPOSITORY=1`；默认 CAS 为 `<output>/.cas`，默认索引为 `<output>/resources.sqlite`，可用 `--import-cas-root` / `BAT_IMPORT_CAS_ROOT` 和 `--import-resource-db` / `BAT_IMPORT_RESOURCE_DB` 覆盖。`resource.index` RPC 可查询现有索引，索引不存在时返回 `available=false`，不会创建空库；`bat doctor cas --output <output>` 或 `bat doctor cas --import-cas-root <path>` 可只读检查既有 CAS 根目录、对象目录、元数据库文件和对象统计。
|
||||
- 官方同步报告中的 `localized_release_status=not_localized` 表示原版资源已发布、汉化资源未发布，这是当前官方同步阶段的正常完成状态；后续 Patch 发布完成后才应切换为 `localized`，表示原版和汉化两套资源都已发布。
|
||||
|
||||
资源同步状态文件默认分布如下：
|
||||
|
||||
@@ -158,6 +158,12 @@ SQLite `ResourceRepository`，索引不存在时返回 `ok=true` 且
|
||||
next attempt；schema 由 `schema_migrations` 版本表管理。
|
||||
- `translation-handoff.json`：当前 release 的版本化 job/unit/provider run 交接
|
||||
快照；worker 更新后的实时状态仍以 `translation-tasks.sqlite` 为准。
|
||||
- `translation-memory.sqlite`：跨 release 的项目级 Translation Memory，不位于
|
||||
`versions/<id>`，也不与 `translation-tasks.sqlite` 共用；记录 raw source/hash、完整
|
||||
TextUnit context、candidate/trusted、translation 和 release/TextUnit/provider/run
|
||||
provenance。默认路径为 `<output>/translation-memory.sqlite`，可由
|
||||
`BAT_TRANSLATION_MEMORY_PATH`、`[translation.worker].translation_memory_path` 或 CLI
|
||||
覆盖。
|
||||
|
||||
删除资源只进入 `official-resource-changes.json`，不进入 Crowdin handoff。
|
||||
|
||||
@@ -209,6 +215,14 @@ SQLite `ResourceRepository`，索引不存在时返回 `ok=true` 且
|
||||
| `translation.task.update` | 已实现 | `{ "task_id": "...", "status": "failed", "failure_reason": "...", "provider_run_id": "..." }` | 写入当前 release 的 provider worker 状态，返回可回查任务记录。 |
|
||||
| `translation.worker.run` | 已实现 | provider worker 参数 | 异步触发 Rust provider worker，返回 `{ "task_id": "...", "kind": "translation.worker.run", "worker": {...} }`。 |
|
||||
| `translation.proofread` | 已实现 | `null` | 将当前汉化 workflow 标记为人工校对中，返回工作流状态报告。 |
|
||||
| `translation.memory.summary` | 已实现 | 可选 `{ "translation_memory_path": "..." }` | 返回 TM schema 版本、总记录数及 candidate/trusted/rejected/superseded 状态计数。 |
|
||||
| `translation.memory.query` | 已实现 | `{ "source_text": "...", "source_context": {...}, "limit": 100 }` | 按 raw source 查询记录，返回 match kind、trust、translation 和 provenance。 |
|
||||
| `translation.memory.confirm` | 已实现 | `{ "record_id": "...", "reviewer": "...", "reason": "..." }` | 显式确认一条 candidate 为 trusted；worker 之后才可自动复用。 |
|
||||
|
||||
TM 的自动复用规则是 raw source 完全相同、完整 context 完全相同且状态为 `trusted`；
|
||||
context 缺失/不一致、normalized source 仅辅助查询、candidate 或 provider 成功都不会
|
||||
自动复用或自动变成 trusted。TM 查询、confirm 和诊断由 Rust `bat` 持有，Go
|
||||
`bat-api` 不维护第二份 TM 状态。
|
||||
|
||||
`parse.status` 是只读查询；没有当前 release 或没有解析缓存时返回
|
||||
`ok=true` 且 `data.available=false`。解析缓存来自官方原版资源目录，不读取
|
||||
@@ -284,6 +298,7 @@ provider worker 参数：
|
||||
| `retry_backoff_seconds` | uint | `5` | 可重试 provider 失败的 next attempt 间隔，可为 0。 |
|
||||
| `max_tasks` | uint/null | `null` | 本轮最多 claim 的任务数，设置时必须大于 0。 |
|
||||
| `worker_id` | string | `bat-rpc-worker` | lease 诊断用 worker ID 前缀。 |
|
||||
| `translation_memory_path` | string/null | 按配置推导 | 覆盖 Rust worker 使用的项目级 TM 数据库路径；未指定时使用 worker 配置或 `<output>/translation-memory.sqlite`。 |
|
||||
|
||||
数字字段必须是 JSON number；字符串数字、负数和越界值会返回
|
||||
`BAT-ERR-700002`。`mock` provider 在没有 fixture 时把 source text 写成可诊断的
|
||||
@@ -299,7 +314,7 @@ token 不会进入报告、任务记录或调试输出。
|
||||
| `localized.publish` | 已实现 | `{ "translation_file": "...", "localized_release_id": "...", "force": false }` 或 `{ "from_worker": true, "localized_release_id": "...", "force": false }` | 已校验并发布的汉化 release、manifest 和完整性报告。 |
|
||||
| `localized.rollback` | 已实现 | `{ "localized_release_id": "..." }`，可省略 | 删除当前 release、恢复 manifest 记录的上一 release 和新状态。 |
|
||||
|
||||
`localized.status` 严格按 daemon / `.env` 中的 `BAT_LOCALIZED_OUTPUT` 或
|
||||
`localized.status` 严格按 daemon / `config.toml` 或环境变量中的 `BAT_LOCALIZED_OUTPUT` 或
|
||||
`--localized-output` 查询汉化产物目录，不把 `./bat-resources` 与
|
||||
`./bat-localized` 混用。当前支持未汉化发布状态和已汉化发布状态的只读报告。
|
||||
`status` / `status_code` 使用生命周期短状态和稳定状态码，例如
|
||||
@@ -432,6 +447,8 @@ CLI 对应关系：
|
||||
| `bat i18n task update` | `translation.task.update` |
|
||||
| `bat i18n worker run` | `translation.worker.run` |
|
||||
| `bat i18n proofread` | `translation.proofread` |
|
||||
| `bat i18n memory summary` / `bat i18n memory query` | `translation.memory.summary` / `translation.memory.query` |
|
||||
| `bat i18n memory confirm` | `translation.memory.confirm` |
|
||||
| `bat localized-status` | `localized.status` |
|
||||
| `bat resource-index` | `resource.index` |
|
||||
|
||||
@@ -463,7 +480,8 @@ CLI 对应关系：
|
||||
`catalog.*`、`parse.*`、
|
||||
`localized.status`、`localized.publish`、`localized.rollback`、
|
||||
`translation.tasks`、`translation.handoff`、`translation.task.update`、
|
||||
`translation.worker.run`、`translation.proofread`、
|
||||
`translation.worker.run`、`translation.proofread`、`translation.memory.summary`、
|
||||
`translation.memory.query`、`translation.memory.confirm`、
|
||||
`task.*` 和三个 `unityfs.patch_*` 方法。
|
||||
- `resource.index` 和 `patch.apply` 当前没有专用 typed helper；需要直接使用 `Call`，并仍须遵守
|
||||
本契约的参数和响应定义。
|
||||
@@ -479,6 +497,7 @@ CLI 对应关系：
|
||||
| `TaskBackend` | `task.list`、`task.status`、`task.logs`、`task.cancel` | 鉴权后的 daemon 任务查询和取消 |
|
||||
| `ParseBackend` | `parse.status`、`parse.text_units`、`parse.errors` | 鉴权后的当前 release 解析状态、TextUnit 和解析错误只读查询 |
|
||||
| `TranslationBackend` | `translation.tasks`、`translation.handoff`、`translation.task.update`、`translation.worker.run`、`translation.proofread` | 鉴权后的 dashboard 翻译任务查询、交接视图、状态回写、provider worker 触发与人工校对标记 |
|
||||
| `TranslationMemoryBackend` | `translation.memory.summary`、`translation.memory.query`、`translation.memory.confirm` | 鉴权后的 TM 摘要、source/context 查询和显式 candidate 确认；Go 只转发，不持有 TM 状态 |
|
||||
| `LocalizedBackend` | `localized.status`、`localized.publish`、`localized.rollback` | 鉴权后的汉化 release 状态、发布与显式回滚 |
|
||||
|
||||
`daemon.stop`、`daemon.clean-stable` 和任意通用 RPC 不属于 bat-api 管理控制面。
|
||||
@@ -487,8 +506,9 @@ Rust dispatch、Go transport 和 bat-api 接口的权威实现位置分别是
|
||||
`internal/api/rpc_release.go`；修改方法、字段或 allowlist 时必须同步更新本文档。
|
||||
|
||||
Go mirror contract fixture 固化在 `internal/api/testdata/contract/`，覆盖
|
||||
`catalog.status` available/unavailable、`resource.manifest` page0 和对应
|
||||
`official-sync-snapshot.json`。这些 fixture 由 Rust 输出归一化而来，只用于
|
||||
`catalog.status` available/unavailable、`resource.manifest` page0、对应
|
||||
`official-sync-snapshot.json` 以及 Translation Memory query/缺库 mirror。
|
||||
这些 fixture/mirror 由 Rust 输出形状归一化而来，只用于
|
||||
schema / mirror 回归；live daemon socket 和完整 fixture release 切换由
|
||||
`make bat-api-local-live-smoke` 在同机 `/tmp` 隔离环境中验证。该 smoke 不替代
|
||||
`make official-smoke` 的官方网络全量下载验证。
|
||||
|
||||
@@ -66,7 +66,8 @@ provider、bundle name、resource type 和 CRC，并有 fixture/golden 回归。
|
||||
|
||||
当前 `cmd/bat-api` 通过 `bat.sock` 读取 Rust 已发布 release，提供 bootstrap、
|
||||
launcher 资源引导兼容、只读 CDN path、readiness、OpenAPI、鉴权管理入口和内嵌
|
||||
dashboard。Rust `bat` 继续拥有资源发现、下载、校验、staging、发布、任务和长期状态。
|
||||
dashboard；翻译任务和 Rust-owned TM 的 summary/query/confirm 也通过 typed RPC
|
||||
转发。Rust `bat` 继续拥有资源发现、下载、校验、staging、发布、任务和长期状态。
|
||||
|
||||
仍需完成：
|
||||
|
||||
@@ -110,9 +111,9 @@ rollback，`localized.status` 能校验当前官方 release 与 patch manifest
|
||||
|
||||
仍需完成通用 patch 发布、复杂重打包、双 release 查询/分发视图和清理策略。
|
||||
|
||||
### G-012：Translation Memory 未实现
|
||||
### G-012：Translation Memory V1 已实现，扩展能力仍缺失
|
||||
|
||||
需要支持精确、模糊和上下文匹配，并保留 provider、模型、审核状态和历史版本。
|
||||
Rust `bat` 已提供独立项目级 SQLite TM，记录 raw source/hash、完整 context、release/TextUnit/provider/run provenance，区分 candidate/trusted，只有显式 confirm 才能建立 trusted 记录；worker 只自动复用 trusted 的 raw source + 完整 context exact match。Go `bat-api` 已提供鉴权的 summary/query 只读接口和 confirm 转发，但 Go 不持有 TM 状态。仍缺少模糊匹配、Glossary 联动和更丰富的导入导出历史能力。
|
||||
|
||||
### G-013：Glossary 未实现
|
||||
|
||||
@@ -144,7 +145,7 @@ rollback，`localized.status` 能校验当前官方 release 与 patch manifest
|
||||
|
||||
1. 继续 G-005：真实 AssetBundle 样本、复杂字段解析和发布级重打包。
|
||||
2. 继续 G-006/G-011D：通用 manifest Patch 和双 release 查询/清理策略。
|
||||
3. 继续 G-011/G-012/G-013/G-014：资源查询、Translation Memory、Glossary 和 Provider
|
||||
3. 继续 G-011/G-012/G-013/G-014：资源查询、TM 扩展、Glossary 和 Provider
|
||||
扩展体系。
|
||||
4. 在隔离环境执行 `make official-smoke`，补充真实网络长期运行报告。
|
||||
5. 最后推进完整 Web 协作后台和完整游戏业务 API。
|
||||
|
||||
@@ -32,7 +32,7 @@
|
||||
| 资源发现 | 读取官方 launcher/resource metadata，解析 `GameMainConfig`、server-info 和 Addressables root | 通过 `bat.sock` 读取已发布版本摘要，不重新探测官方 metadata |
|
||||
| 下载与发布 | 下载、校验、staging、原子发布 `current -> versions/<id>`，维护 manifest/snapshot/version-state | 不下载、不写 staging、不改 version-state；生产资源根来自 RPC 返回的 `resource_root` |
|
||||
| 启动前资源入口 | 暴露 `catalog.status` / `resource.manifest` 等 RPC 数据 | 提供 `/v1/bootstrap`、`/v1/launcher/bootstrap`、launcher 资源 metadata 兼容端点、`/v1/server-info`、CDN path 和 `/admin/dashboard/`，组织给客户端/补丁器/维护者使用 |
|
||||
| 长期状态 | watch/daemon、任务队列、日志、错误码、repair/sync/verify | 周期性经 RPC 刷新内存索引；认证 Web 控制面仅白名单转发 reload/refresh/restart/sync/verify/repair/catalog-refresh、schedule/task/log/parse/translation/localized 方法，不持有或写入同步状态 |
|
||||
| 长期状态 | watch/daemon、任务队列、日志、错误码、repair/sync/verify | 周期性经 RPC 刷新内存索引；认证 Web 控制面仅白名单转发 reload/refresh/restart/sync/verify/repair/catalog-refresh、schedule/task/log/parse/translation/TM/localized 方法，不持有或写入同步状态 |
|
||||
|
||||
这条边界允许 `bat-api` 做资源 bootstrap 兼容，但不允许它复制 Rust 下载器或伪装完整游戏业务服务。
|
||||
|
||||
@@ -95,8 +95,8 @@
|
||||
| 组件 | 路径 | 状态 | 说明 |
|
||||
|---|---|---|---|
|
||||
| Module | `go.mod` → `bat-api` | 已用 | 服务层模块名 |
|
||||
| RPC client | `internal/backendrpc` | **完成** | Unix socket JSON-RPC transport + typed helper；typed helper 覆盖 daemon 已实现控制/查询、`resource.state/sync/verify/repair/manifest/list`、`catalog.*`、`parse.*`、`localized.status/publish/rollback`、`task.*`、`translation.tasks`、`translation.handoff`、`translation.task.update`、`translation.worker.run`、`translation.proofread` 和文件级 UnityFS patch 调用；`resource.index`、`patch.apply` 仍通过通用 `Call` 走同一 contract；fake transport 单测，配合 `internal/api/testdata/contract/` 固化 Rust 输出 mirror |
|
||||
| 资源 bootstrap/分发 | `cmd/bat-api` + `internal/api` | **MVP+生产控制面** | RPC 发现 + 周期刷新/诊断 + `/v1/bootstrap` + `/v1/launcher/bootstrap` + launcher 资源 metadata 兼容 + `/readyz` + CDN Range/缓存头 + 鉴权/限流/访问日志/反代适配 + OpenAPI + 管理控制白名单 + 内嵌 dashboard + `.env` |
|
||||
| RPC client | `internal/backendrpc` | **完成** | Unix socket JSON-RPC transport + typed helper；typed helper 覆盖 daemon 已实现控制/查询、`resource.state/sync/verify/repair/manifest/list`、`catalog.*`、`parse.*`、`localized.status/publish/rollback`、`task.*`、`translation.tasks`、`translation.handoff`、`translation.task.update`、`translation.worker.run`、`translation.proofread`、`translation.memory.summary/query/confirm` 和文件级 UnityFS patch 调用；`resource.index`、`patch.apply` 仍通过通用 `Call` 走同一 contract；fake transport 单测和 `internal/api/testdata/contract/` mirror test 固化 Rust 输出字段 |
|
||||
| 资源 bootstrap/分发 | `cmd/bat-api` + `internal/api` | **MVP+生产控制面** | RPC 发现 + 周期刷新/诊断 + `/v1/bootstrap` + `/v1/launcher/bootstrap` + launcher 资源 metadata 兼容 + `/readyz` + CDN Range/缓存头 + 鉴权/限流/访问日志/反代适配 + OpenAPI + 管理控制白名单 + translation/TM admin forwarding + 内嵌 dashboard + `.env` |
|
||||
| 试验 CLI | `cmd/bat` | **试验** | doctor 固定 ok；manifest/sync 走 FFI |
|
||||
| FFI | `internal/ffi` | **可选** | 需 `build-ffi` |
|
||||
| 空骨架 | `api/`、`pkg/*`、部分 `internal/*` | **空** | 见各目录 README |
|
||||
@@ -134,7 +134,7 @@ make build-go-cli # 产出 bin/bat-go
|
||||
| 项 | 状态 |
|
||||
|---|---|
|
||||
| Go 同步 CLI | **边界已确定**（正式同步 CLI = Rust `bat`） |
|
||||
| G-009 bat-api 资源 bootstrap/分发 | **资源面完成（非完整官方游戏 API）**；已含资源 bootstrap、launcher resource metadata 兼容、HTTP 鉴权/限流/日志/反代适配、RPC 周期刷新/诊断、readiness、OpenAPI、管理控制白名单、Rust-owned `schedule.*`、`task.*`、`parse.*`、翻译状态回写代理、内嵌 dashboard、同机 live smoke 和部署模板；持久化仍另议 |
|
||||
| G-009 bat-api 资源 bootstrap/分发 | **资源面完成（非完整官方游戏 API）**；已含资源 bootstrap、launcher resource metadata 兼容、HTTP 鉴权/限流/日志/反代适配、RPC 周期刷新/诊断、readiness、OpenAPI、管理控制白名单、Rust-owned `schedule.*`、`task.*`、`parse.*`、翻译任务/TM 状态查询与显式确认代理、内嵌 dashboard、同机 live smoke 和部署模板；持久化仍另议 |
|
||||
| G-010 Web | 内嵌 dashboard MVP 已完成；完整协作后台、登录/角色、术语管理和构建型前端未开始 |
|
||||
|
||||
---
|
||||
|
||||
+653
-208
File diff suppressed because it is too large
Load Diff
@@ -17,9 +17,11 @@ fn parse_with_env(values: &[&str], env: &[(&str, &str)]) -> anyhow::Result<CliOp
|
||||
.iter()
|
||||
.map(|(key, value)| (key.to_string(), value.to_string()))
|
||||
.collect();
|
||||
parse_args_with_env(values.iter().map(|value| value.to_string()), move |key| {
|
||||
map.get(key).cloned()
|
||||
})
|
||||
parse_args_with_env(
|
||||
values.iter().map(|value| value.to_string()),
|
||||
move |key| map.get(key).cloned(),
|
||||
None,
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -105,7 +107,7 @@ fn env_watch_daemon_only_affect_bare_run() {
|
||||
|
||||
#[test]
|
||||
fn env_values_do_not_break_status_and_reload_guard() {
|
||||
// .env 提供的代理/工具/输出目录不算"显式同步参数"，status 应照常可用。
|
||||
// 配置文件/环境变量提供的代理/工具/输出目录不算"显式同步参数"，status 应照常可用。
|
||||
let options = parse_with_env(
|
||||
&["bat", "status"],
|
||||
&[
|
||||
@@ -222,6 +224,148 @@ fn translation_worker_command_options_are_validated() {
|
||||
assert!(parse(&["bat", "i18n", "worker", "run", "--provider", "unknown"]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn translation_memory_commands_parse_and_validate() {
|
||||
let summary = parse(&["bat", "i18n", "memory", "summary"]).unwrap();
|
||||
assert_eq!(summary.command, CliCommand::TranslationMemorySummary);
|
||||
let query = parse(&[
|
||||
"bat",
|
||||
"i18n",
|
||||
"tm",
|
||||
"query",
|
||||
"--tm-source-text",
|
||||
"Hello",
|
||||
"--tm-context-json",
|
||||
r#"{"destination":"story.bundle"}"#,
|
||||
"--limit",
|
||||
"5",
|
||||
])
|
||||
.unwrap();
|
||||
assert_eq!(query.command, CliCommand::TranslationMemoryQuery);
|
||||
assert_eq!(
|
||||
query.translation_memory_source_text.as_deref(),
|
||||
Some("Hello")
|
||||
);
|
||||
let confirm = parse(&[
|
||||
"bat",
|
||||
"i18n",
|
||||
"memory",
|
||||
"confirm",
|
||||
"--tm-record-id",
|
||||
"tm-record",
|
||||
"--tm-reviewer",
|
||||
"reviewer",
|
||||
"--tm-reason",
|
||||
"accepted",
|
||||
])
|
||||
.unwrap();
|
||||
assert_eq!(confirm.command, CliCommand::TranslationMemoryConfirm);
|
||||
assert!(parse(&["bat", "i18n", "memory", "query"]).is_err());
|
||||
assert!(parse(&[
|
||||
"bat",
|
||||
"i18n",
|
||||
"memory",
|
||||
"confirm",
|
||||
"--tm-record-id",
|
||||
"tm-record"
|
||||
])
|
||||
.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn restart_reload_accept_translation_worker_startup_options() {
|
||||
let restart = parse(&[
|
||||
"bat",
|
||||
"restart",
|
||||
"--translation-provider",
|
||||
"mock",
|
||||
"--translation-memory-path",
|
||||
"/tmp/tm.sqlite",
|
||||
"--worker-concurrency",
|
||||
"2",
|
||||
])
|
||||
.unwrap();
|
||||
assert_eq!(restart.command, CliCommand::Restart);
|
||||
assert!(restart.translation_worker_option_explicit);
|
||||
assert_eq!(
|
||||
restart.translation_memory_path,
|
||||
Some(PathBuf::from("/tmp/tm.sqlite"))
|
||||
);
|
||||
assert_eq!(restart.worker_concurrency, 2);
|
||||
assert!(restart.config.auto_discover);
|
||||
|
||||
let reload = parse(&["bat", "reload", "--worker-id", "reload-worker"]).unwrap();
|
||||
assert_eq!(reload.command, CliCommand::Reload);
|
||||
assert!(reload.translation_worker_option_explicit);
|
||||
assert_eq!(reload.worker_id.as_deref(), Some("reload-worker"));
|
||||
assert!(reload.config.auto_discover);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn translation_memory_command_options_do_not_apply_to_other_commands() {
|
||||
let error = parse(&["bat", "restart", "--tm-source-text", "Hello"]).unwrap_err();
|
||||
assert!(error.to_string().contains("查询/confirm"));
|
||||
|
||||
let error = parse(&["bat", "status", "--tm-record-id", "tm-record"]).unwrap_err();
|
||||
assert!(error.to_string().contains("查询/confirm"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn translation_memory_subcommands_reject_irrelevant_options() {
|
||||
let error = parse(&[
|
||||
"bat",
|
||||
"i18n",
|
||||
"memory",
|
||||
"summary",
|
||||
"--tm-source-text",
|
||||
"Hello",
|
||||
])
|
||||
.unwrap_err();
|
||||
assert!(error.to_string().contains("summary"));
|
||||
let error = parse(&["bat", "i18n", "memory", "summary", "--limit", "5"]).unwrap_err();
|
||||
assert!(error.to_string().contains("summary"));
|
||||
|
||||
let error = parse(&[
|
||||
"bat",
|
||||
"i18n",
|
||||
"memory",
|
||||
"query",
|
||||
"--tm-source-text",
|
||||
"Hello",
|
||||
"--tm-record-id",
|
||||
"tm-record",
|
||||
])
|
||||
.unwrap_err();
|
||||
assert!(error.to_string().contains("confirm 参数"));
|
||||
let error = parse(&[
|
||||
"bat",
|
||||
"i18n",
|
||||
"memory",
|
||||
"query",
|
||||
"--tm-source-text",
|
||||
"Hello",
|
||||
"--offset",
|
||||
"1",
|
||||
])
|
||||
.unwrap_err();
|
||||
assert!(error.to_string().contains("查询过滤"));
|
||||
|
||||
let error = parse(&[
|
||||
"bat",
|
||||
"i18n",
|
||||
"memory",
|
||||
"confirm",
|
||||
"--tm-record-id",
|
||||
"tm-record",
|
||||
"--tm-reviewer",
|
||||
"reviewer",
|
||||
"--tm-context-json",
|
||||
"{}",
|
||||
])
|
||||
.unwrap_err();
|
||||
assert!(error.to_string().contains("query 参数"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn translation_worker_env_defaults_apply() {
|
||||
let options = parse_with_env(
|
||||
@@ -229,6 +373,7 @@ fn translation_worker_env_defaults_apply() {
|
||||
&[
|
||||
("BAT_TRANSLATION_PROVIDER", "mock"),
|
||||
("BAT_TRANSLATION_FIXTURE", "/tmp/fixture.json"),
|
||||
("BAT_TRANSLATION_MEMORY_PATH", "/tmp/tm.sqlite"),
|
||||
("BAT_TRANSLATION_CONCURRENCY", "16"),
|
||||
("BAT_TRANSLATION_MAX_ATTEMPTS", "5"),
|
||||
("BAT_TRANSLATION_LEASE_SECONDS", "120"),
|
||||
@@ -244,6 +389,10 @@ fn translation_worker_env_defaults_apply() {
|
||||
options.translation_fixture,
|
||||
Some(PathBuf::from("/tmp/fixture.json"))
|
||||
);
|
||||
assert_eq!(
|
||||
options.translation_memory_path,
|
||||
Some(PathBuf::from("/tmp/tm.sqlite"))
|
||||
);
|
||||
assert_eq!(options.worker_concurrency, 16);
|
||||
assert_eq!(options.worker_max_attempts, 5);
|
||||
assert_eq!(options.worker_lease_seconds, 120);
|
||||
@@ -610,6 +759,8 @@ fn translation_workbench_commands_read_update_and_clear_entries() {
|
||||
translated_text: None,
|
||||
translation_provider: None,
|
||||
provider_run_id: None,
|
||||
translation_source_kind: None,
|
||||
translation_memory_record_id: None,
|
||||
translated_unix_seconds: None,
|
||||
review_status: None,
|
||||
format: Some("plain".to_string()),
|
||||
@@ -1030,46 +1181,72 @@ fn cli_download_concurrency_is_preserved_for_daemon_child() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_env_line_handles_quotes_and_rejects_bad_keys() {
|
||||
assert_eq!(
|
||||
parse_env_line("KEY=value"),
|
||||
Some(("KEY".to_string(), "value".to_string()))
|
||||
);
|
||||
assert_eq!(
|
||||
parse_env_line("KEY=\"quoted value\""),
|
||||
Some(("KEY".to_string(), "quoted value".to_string()))
|
||||
);
|
||||
assert_eq!(
|
||||
parse_env_line("KEY='single'"),
|
||||
Some(("KEY".to_string(), "single".to_string()))
|
||||
);
|
||||
assert_eq!(
|
||||
parse_env_line("BAT_OUTPUT = ./x"),
|
||||
Some(("BAT_OUTPUT".to_string(), "./x".to_string()))
|
||||
);
|
||||
assert_eq!(parse_env_line("no_equals_sign"), None);
|
||||
assert_eq!(parse_env_line("1BAD=x"), None);
|
||||
assert_eq!(parse_env_line("BAD KEY=x"), None);
|
||||
}
|
||||
fn config_file_is_applied_before_env_and_cli() {
|
||||
let temp = tempfile::TempDir::new().unwrap();
|
||||
let config_path = temp.path().join(super::config_file::CONFIG_FILE_NAME);
|
||||
std::fs::write(
|
||||
&config_path,
|
||||
r#"
|
||||
[runtime]
|
||||
state_dir = '/srv/state'
|
||||
output_format = 'json'
|
||||
|
||||
#[test]
|
||||
fn env_template_is_parseable_and_bootstrap_ready() {
|
||||
// 模板每个非注释行必须可解析；无参启动所需的最小配置默认启用。
|
||||
let mut keys = Vec::new();
|
||||
for line in ENV_TEMPLATE.lines() {
|
||||
let line = line.trim();
|
||||
if line.is_empty() || line.starts_with('#') {
|
||||
continue;
|
||||
}
|
||||
let (key, _) = parse_env_line(line).unwrap_or_else(|| panic!("模板行必须可解析：{line}"));
|
||||
keys.push(key);
|
||||
[resource]
|
||||
output_root = '/srv/from-config'
|
||||
auto_discover = true
|
||||
|
||||
[localized]
|
||||
output_root = '/srv/from-config-localized'
|
||||
|
||||
[network]
|
||||
proxy = 'none'
|
||||
download_concurrency = 12
|
||||
|
||||
[translation.worker]
|
||||
translation_memory_path = '/srv/config-tm.sqlite'
|
||||
"#,
|
||||
)
|
||||
.unwrap();
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
let mut permissions = std::fs::metadata(&config_path).unwrap().permissions();
|
||||
permissions.set_mode(0o600);
|
||||
std::fs::set_permissions(&config_path, permissions).unwrap();
|
||||
}
|
||||
assert!(keys.contains(&"BAT_OUTPUT".to_string()));
|
||||
assert!(keys.contains(&"BAT_LOCALIZED_OUTPUT".to_string()));
|
||||
assert!(keys.contains(&"BAT_IMPORT_REPOSITORY".to_string()));
|
||||
assert!(keys.contains(&"BAT_IMPORT_CAS_ROOT".to_string()));
|
||||
assert!(keys.contains(&"BAT_IMPORT_RESOURCE_DB".to_string()));
|
||||
assert!(keys.contains(&"BAT_AUTO_DISCOVER".to_string()));
|
||||
let config = super::config_file::load_from_binary_dir(temp.path())
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
let options = parse_args_with_env(
|
||||
vec![
|
||||
"bat".to_string(),
|
||||
"--daemon".to_string(),
|
||||
"--output".to_string(),
|
||||
"/cli/output".to_string(),
|
||||
"--translation-memory-path".to_string(),
|
||||
"/cli/tm.sqlite".to_string(),
|
||||
],
|
||||
|key| match key {
|
||||
"BAT_OUTPUT" => Some("/env/output".to_string()),
|
||||
"BAT_TRANSLATION_MEMORY_PATH" => Some("/env/tm.sqlite".to_string()),
|
||||
_ => None,
|
||||
},
|
||||
Some(&config),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(options.state_dir, PathBuf::from("/srv/state"));
|
||||
assert_eq!(options.output_format, OutputFormat::Json);
|
||||
assert_eq!(options.config.output_root, PathBuf::from("/cli/output"));
|
||||
assert_eq!(
|
||||
options.config.localized_output_root,
|
||||
PathBuf::from("/srv/from-config-localized")
|
||||
);
|
||||
assert_eq!(options.config.download_concurrency, 12);
|
||||
assert_eq!(options.config.curl_proxy.mode(), &CurlProxyMode::Disabled);
|
||||
assert_eq!(
|
||||
options.translation_memory_path,
|
||||
Some(PathBuf::from("/cli/tm.sqlite"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1326,6 +1503,14 @@ fn rejects_proxy_with_unsupported_scheme() {
|
||||
assert!(parse(&["bat", "--proxy", "127.0.0.1:7890"]).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn proxy_parse_errors_redact_credentials() {
|
||||
let error = parse(&["bat", "--proxy", "http://user:secret@"]).unwrap_err();
|
||||
let message = error.to_string();
|
||||
assert!(!message.contains("secret"));
|
||||
assert!(message.contains("<redacted>"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_watch_defaults_to_one_hour_and_quiet_up_to_date() {
|
||||
let options = parse(&["bat", "--auto-discover", "--watch", "--interval", "30m"]).unwrap();
|
||||
@@ -1997,6 +2182,54 @@ fn daemon_child_args_preserve_sync_options() {
|
||||
assert!(args.contains(&"--no-banner".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn daemon_child_args_preserve_translation_worker_configuration() {
|
||||
let options = parse(&[
|
||||
"bat",
|
||||
"--daemon",
|
||||
"--output",
|
||||
"/tmp/daemon-output",
|
||||
"--translation-provider",
|
||||
"mock",
|
||||
"--translation-fixture",
|
||||
"/tmp/provider.json",
|
||||
"--translation-memory-path",
|
||||
"/tmp/tm.sqlite",
|
||||
"--worker-concurrency",
|
||||
"4",
|
||||
"--worker-max-attempts",
|
||||
"5",
|
||||
"--worker-lease-seconds",
|
||||
"60",
|
||||
"--worker-retry-backoff-seconds",
|
||||
"2",
|
||||
"--worker-max-tasks",
|
||||
"3",
|
||||
"--worker-id",
|
||||
"daemon-worker",
|
||||
])
|
||||
.unwrap();
|
||||
let args = daemon_child_args(&options);
|
||||
let mut child_args = vec!["bat".to_string()];
|
||||
child_args.extend(args);
|
||||
let child = parse_args_from(child_args).unwrap();
|
||||
assert_eq!(child.translation_provider.as_deref(), Some("mock"));
|
||||
assert_eq!(
|
||||
child.translation_fixture,
|
||||
Some(PathBuf::from("/tmp/provider.json"))
|
||||
);
|
||||
assert_eq!(
|
||||
child.translation_memory_path,
|
||||
Some(PathBuf::from("/tmp/tm.sqlite"))
|
||||
);
|
||||
assert_eq!(child.worker_concurrency, 4);
|
||||
assert_eq!(child.worker_max_attempts, 5);
|
||||
assert_eq!(child.worker_lease_seconds, 60);
|
||||
assert_eq!(child.worker_retry_backoff, Duration::from_secs(2));
|
||||
assert_eq!(child.worker_max_tasks, Some(3));
|
||||
assert_eq!(child.worker_id.as_deref(), Some("daemon-worker"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn curl_proxy_url_extracts_only_url_mode() {
|
||||
assert_eq!(
|
||||
@@ -2184,6 +2417,7 @@ fn test_task_context_with_config(base_config: OfficialUpdateConfig) -> DaemonTas
|
||||
registry: TaskRegistry::new(),
|
||||
queue,
|
||||
base_config,
|
||||
translation_worker_config: TranslationWorkerConfig::default(),
|
||||
sync_lock: Arc::new(Mutex::new(())),
|
||||
restart_controller: test_restart_controller,
|
||||
}
|
||||
@@ -2354,6 +2588,7 @@ fn dispatch_daemon_doctor_returns_report() {
|
||||
registry: TaskRegistry::new(),
|
||||
queue,
|
||||
base_config,
|
||||
translation_worker_config: TranslationWorkerConfig::default(),
|
||||
sync_lock: Arc::new(Mutex::new(())),
|
||||
restart_controller: test_restart_controller,
|
||||
};
|
||||
@@ -2374,6 +2609,112 @@ fn dispatch_daemon_doctor_returns_report() {
|
||||
assert!(checks.iter().any(|check| check["name"] == "daemon_rpc"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dispatch_translation_memory_summary_reports_missing_database_without_creating_it() {
|
||||
let temp = tempfile::TempDir::new().unwrap();
|
||||
let output_root = temp.path().join("output");
|
||||
let state_dir = temp.path().join("state");
|
||||
let (queue, _rx) = mpsc::channel::<TaskJob>();
|
||||
let context = DaemonTaskContext {
|
||||
registry: TaskRegistry::new(),
|
||||
queue,
|
||||
base_config: OfficialUpdateConfig {
|
||||
output_root: output_root.clone(),
|
||||
..Default::default()
|
||||
},
|
||||
translation_worker_config: TranslationWorkerConfig::default(),
|
||||
sync_lock: Arc::new(Mutex::new(())),
|
||||
restart_controller: test_restart_controller,
|
||||
};
|
||||
|
||||
let envelope = dispatch_rpc_method(
|
||||
&rpc_request("translation.memory.summary", None),
|
||||
&state_dir,
|
||||
&new_daemon_control(),
|
||||
&context,
|
||||
"req-tm-summary-1".to_string(),
|
||||
);
|
||||
let value = serde_json::to_value(envelope).unwrap();
|
||||
assert_eq!(value["ok"], true);
|
||||
assert_eq!(value["data"]["available"], false);
|
||||
assert_eq!(value["data"]["reason"], "database_missing");
|
||||
assert!(!output_root.join("translation-memory.sqlite").exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dispatch_translation_memory_summary_defaults_to_worker_config_path() {
|
||||
let temp = tempfile::TempDir::new().unwrap();
|
||||
let output_root = temp.path().join("output");
|
||||
let state_dir = temp.path().join("state");
|
||||
let configured_tm_path = temp.path().join("configured-tm.sqlite");
|
||||
let (queue, _rx) = mpsc::channel::<TaskJob>();
|
||||
let context = DaemonTaskContext {
|
||||
registry: TaskRegistry::new(),
|
||||
queue,
|
||||
base_config: OfficialUpdateConfig {
|
||||
output_root,
|
||||
..Default::default()
|
||||
},
|
||||
translation_worker_config: TranslationWorkerConfig {
|
||||
translation_memory_path: Some(configured_tm_path.clone()),
|
||||
..TranslationWorkerConfig::default()
|
||||
},
|
||||
sync_lock: Arc::new(Mutex::new(())),
|
||||
restart_controller: test_restart_controller,
|
||||
};
|
||||
|
||||
let envelope = dispatch_rpc_method(
|
||||
&rpc_request("translation.memory.summary", None),
|
||||
&state_dir,
|
||||
&new_daemon_control(),
|
||||
&context,
|
||||
"req-tm-summary-configured-1".to_string(),
|
||||
);
|
||||
let value = serde_json::to_value(envelope).unwrap();
|
||||
assert_eq!(value["ok"], true);
|
||||
assert_eq!(value["data"]["available"], false);
|
||||
assert_eq!(
|
||||
value["data"]["path"].as_str(),
|
||||
Some(configured_tm_path.to_str().unwrap())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dispatch_translation_memory_rejects_invalid_params_with_stable_error_code() {
|
||||
let temp = tempfile::TempDir::new().unwrap();
|
||||
let context = test_task_context_with_config(OfficialUpdateConfig {
|
||||
output_root: temp.path().join("output"),
|
||||
..Default::default()
|
||||
});
|
||||
|
||||
for (method, params) in [
|
||||
("translation.memory.query", None),
|
||||
(
|
||||
"translation.memory.query",
|
||||
Some(serde_json::json!({ "source_text": "Hello", "limit": "1" })),
|
||||
),
|
||||
(
|
||||
"translation.memory.summary",
|
||||
Some(serde_json::json!({ "translation_memory_path": 42 })),
|
||||
),
|
||||
(
|
||||
"translation.memory.confirm",
|
||||
Some(serde_json::json!({ "record_id": "tm-record", "reviewer": 42 })),
|
||||
),
|
||||
] {
|
||||
let envelope = dispatch_rpc_method(
|
||||
&rpc_request(method, params),
|
||||
temp.path(),
|
||||
&new_daemon_control(),
|
||||
&context,
|
||||
format!("req-invalid-{method}"),
|
||||
);
|
||||
let value = serde_json::to_value(envelope).unwrap();
|
||||
assert_eq!(value["ok"], false, "method={method}");
|
||||
assert_eq!(value["error"]["code"], "BAT-ERR-700002", "method={method}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dispatch_resource_sync_enqueues_task() {
|
||||
let temp = tempfile::TempDir::new().unwrap();
|
||||
@@ -2384,6 +2725,7 @@ fn dispatch_resource_sync_enqueues_task() {
|
||||
registry: TaskRegistry::new(),
|
||||
queue,
|
||||
base_config: OfficialUpdateConfig::default(),
|
||||
translation_worker_config: TranslationWorkerConfig::default(),
|
||||
sync_lock: Arc::new(Mutex::new(())),
|
||||
restart_controller: test_restart_controller,
|
||||
};
|
||||
@@ -2445,6 +2787,7 @@ fn dispatch_resource_repair_enqueues_repair_task() {
|
||||
registry: TaskRegistry::new(),
|
||||
queue,
|
||||
base_config,
|
||||
translation_worker_config: TranslationWorkerConfig::default(),
|
||||
sync_lock: Arc::new(Mutex::new(())),
|
||||
restart_controller: test_restart_controller,
|
||||
};
|
||||
@@ -3737,6 +4080,7 @@ fn dispatch_catalog_refresh_enqueues_task() {
|
||||
registry: TaskRegistry::new(),
|
||||
queue,
|
||||
base_config: OfficialUpdateConfig::default(),
|
||||
translation_worker_config: TranslationWorkerConfig::default(),
|
||||
sync_lock: Arc::new(Mutex::new(())),
|
||||
restart_controller: test_restart_controller,
|
||||
};
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -111,6 +111,13 @@ impl HumanReport for bat_infrastructure::TranslationWorkerReport {
|
||||
print_field("失败任务", self.failed_count);
|
||||
print_field("已安排重试", self.retry_scheduled_count);
|
||||
print_field("剩余任务", self.remaining_count);
|
||||
print_path_field("Translation Memory", &self.translation_memory_path);
|
||||
print_field("TM 可用", format_bool(self.translation_memory_available));
|
||||
print_field("TM 命中 TextUnit", self.translation_memory_hit_count);
|
||||
print_field("Provider TextUnit", self.provider_unit_count);
|
||||
for failure in &self.translation_memory_failures {
|
||||
println!(" - TM: {failure}");
|
||||
}
|
||||
for failure in &self.failures {
|
||||
println!(
|
||||
" - {} [{}] retryable={} {}",
|
||||
|
||||
@@ -519,6 +519,8 @@ pub(super) struct DaemonTaskContext {
|
||||
pub(super) registry: TaskRegistry,
|
||||
pub(super) queue: mpsc::Sender<TaskJob>,
|
||||
pub(super) base_config: OfficialUpdateConfig,
|
||||
/// daemon 中未显式传入参数的 translation worker 默认配置。
|
||||
pub(super) translation_worker_config: TranslationWorkerConfig,
|
||||
/// 串行化会读取或修改已发布资源状态的 daemon 操作。
|
||||
pub(super) sync_lock: Arc<Mutex<()>>,
|
||||
pub(super) restart_controller: DaemonRestartController,
|
||||
@@ -562,11 +564,15 @@ pub(super) fn run_task_worker(
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let cancel_check = Arc::clone(&cancel);
|
||||
runtime
|
||||
.block_on(bat_infrastructure::run_translation_worker_at(
|
||||
&resource_root,
|
||||
worker_config,
|
||||
))
|
||||
.block_on(
|
||||
bat_infrastructure::run_translation_worker_at_with_cancellation(
|
||||
&resource_root,
|
||||
worker_config,
|
||||
Arc::new(move || cancel_check.load(Ordering::Relaxed)),
|
||||
),
|
||||
)
|
||||
.and_then(|report| serde_json::to_value(report).map_err(anyhow::Error::from))
|
||||
})
|
||||
} else {
|
||||
@@ -608,7 +614,8 @@ pub(super) fn run_task_worker(
|
||||
record.result = Some(report);
|
||||
}),
|
||||
Err(error) => {
|
||||
let cancelled = cancel.load(Ordering::Relaxed);
|
||||
let cancelled =
|
||||
cancel.load(Ordering::Relaxed) || daemon_control_stop_requested(Some(&control));
|
||||
// 下载失败携带类型化 DownloadError（含准确网络域码）；其余归 internal。
|
||||
let code = error
|
||||
.downcast_ref::<bat_infrastructure::DownloadError>()
|
||||
|
||||
@@ -71,28 +71,24 @@ pub(super) fn print_startup_banner() {
|
||||
eprintln!("{STARTUP_BANNER}");
|
||||
}
|
||||
|
||||
pub(super) fn print_env_template_created(path: &Path) {
|
||||
pub(super) fn print_config_template_created(path: &Path) {
|
||||
eprintln!(
|
||||
"已生成配置模板 {}（编辑其中的 BAT_* 配置后，直接运行 `bat` 即可按 .env 启动）",
|
||||
"已生成配置模板 {}（编辑 `config.toml`，`config.toml.example` 不会被程序自动读取）",
|
||||
path.display()
|
||||
);
|
||||
}
|
||||
|
||||
pub(super) fn print_env_template_warning(path: &Path, error: impl std::fmt::Display) {
|
||||
eprintln!("警告：生成 .env 配置模板失败 {}：{error}", path.display());
|
||||
}
|
||||
|
||||
pub(super) fn print_env_read_warning(path: &Path, error: impl std::fmt::Display) {
|
||||
eprintln!("警告：读取 .env 失败 {}：{error}", path.display());
|
||||
}
|
||||
|
||||
pub(super) fn print_env_parse_warning(line_number: usize, raw_line: &str) {
|
||||
pub(super) fn print_config_template_warning(path: &Path, error: impl std::fmt::Display) {
|
||||
eprintln!(
|
||||
"警告：.env 第 {} 行无法解析，已忽略：{raw_line}",
|
||||
line_number
|
||||
"警告：生成 config.toml.example 模板失败 {}：{error}",
|
||||
path.display()
|
||||
);
|
||||
}
|
||||
|
||||
pub(super) fn print_deprecated_env_file_warning() {
|
||||
eprintln!("警告：BAT_SKIP_ENV_FILE 已废弃且不再影响启动，已忽略");
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub(super) struct ProgressLogger {
|
||||
enabled: bool,
|
||||
|
||||
@@ -1,4 +1,8 @@
|
||||
use super::report_output::print_json_value;
|
||||
use super::*;
|
||||
use bat_core::domain::TranslationMemoryContext;
|
||||
use bat_core::repositories::TranslationMemoryRepository;
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct TranslationTaskResultUpdateParam {
|
||||
@@ -332,6 +336,8 @@ fn build_manual_translation_results(
|
||||
unit_id: unit_id.to_string(),
|
||||
source_text: param.source_text.clone(),
|
||||
translated_text: param.translated_text.clone(),
|
||||
source_kind: bat_infrastructure::TranslationTaskResultSourceKind::Manual,
|
||||
translation_memory_record_id: None,
|
||||
provider: provider.to_string(),
|
||||
provider_run_id: provider_run_id.to_string(),
|
||||
translated_unix_seconds,
|
||||
@@ -355,3 +361,241 @@ pub(super) fn translation_task_query_json(query: &OfficialTextUnitTaskQuery) ->
|
||||
"has_failure_reason": query.has_failure_reason,
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) fn run_translation_memory_command(options: &CliOptions) -> anyhow::Result<()> {
|
||||
let method = match options.command {
|
||||
CliCommand::TranslationMemorySummary => RPC_METHOD_TRANSLATION_MEMORY_SUMMARY,
|
||||
CliCommand::TranslationMemoryQuery => RPC_METHOD_TRANSLATION_MEMORY_QUERY,
|
||||
CliCommand::TranslationMemoryConfirm => RPC_METHOD_TRANSLATION_MEMORY_CONFIRM,
|
||||
_ => return Err(anyhow::anyhow!("不是 Translation Memory 命令")),
|
||||
};
|
||||
if daemon_rpc_available(&options.state_dir)
|
||||
&& options.resource_root.is_none()
|
||||
&& !options.output_explicit
|
||||
{
|
||||
let _control_lock = DaemonControlLock::acquire(&options.state_dir)?;
|
||||
let report = daemon_rpc_call(
|
||||
&options.state_dir,
|
||||
method,
|
||||
translation_memory_cli_params(options)?,
|
||||
)?;
|
||||
print_json_value(options.output_format, &report)?;
|
||||
return Ok(());
|
||||
}
|
||||
let path = translation_memory_cli_path(options)?;
|
||||
let report = match options.command {
|
||||
CliCommand::TranslationMemorySummary => build_translation_memory_summary_report(&path)?,
|
||||
CliCommand::TranslationMemoryQuery => {
|
||||
let source_text = options
|
||||
.translation_memory_source_text
|
||||
.as_deref()
|
||||
.ok_or_else(|| anyhow::anyhow!("TM query 必须指定 --tm-source-text"))?;
|
||||
let context = parse_translation_memory_context(
|
||||
options.translation_memory_context_json.as_deref(),
|
||||
)?;
|
||||
build_translation_memory_query_report(
|
||||
&path,
|
||||
source_text,
|
||||
&context,
|
||||
options.query_limit,
|
||||
)?
|
||||
}
|
||||
CliCommand::TranslationMemoryConfirm => {
|
||||
let record_id = options
|
||||
.translation_memory_record_id
|
||||
.as_deref()
|
||||
.ok_or_else(|| anyhow::anyhow!("TM confirm 必须指定 --tm-record-id"))?;
|
||||
let reviewer = options
|
||||
.translation_memory_reviewer
|
||||
.as_deref()
|
||||
.ok_or_else(|| anyhow::anyhow!("TM confirm 必须指定 --tm-reviewer"))?;
|
||||
build_translation_memory_confirm_report(
|
||||
&path,
|
||||
record_id,
|
||||
reviewer,
|
||||
options.translation_memory_reason.clone(),
|
||||
)?
|
||||
}
|
||||
_ => unreachable!(),
|
||||
};
|
||||
print_json_value(options.output_format, &report)
|
||||
}
|
||||
|
||||
fn translation_memory_cli_path(options: &CliOptions) -> anyhow::Result<std::path::PathBuf> {
|
||||
if let Some(path) = options.translation_memory_path.as_ref() {
|
||||
return lexical_absolute(path).map_err(anyhow::Error::msg);
|
||||
}
|
||||
let resource_root = options
|
||||
.resource_root
|
||||
.as_deref()
|
||||
.map(lexical_absolute)
|
||||
.transpose()
|
||||
.map_err(anyhow::Error::msg)?
|
||||
.unwrap_or(active_official_resource_root(&options.config.output_root)?);
|
||||
Ok(bat_infrastructure::translation_memory_repository_path(
|
||||
&resource_root,
|
||||
))
|
||||
}
|
||||
|
||||
fn translation_memory_cli_params(
|
||||
options: &CliOptions,
|
||||
) -> anyhow::Result<Option<serde_json::Value>> {
|
||||
let mut params = serde_json::Map::new();
|
||||
if let Some(path) = options.translation_memory_path.as_ref() {
|
||||
params.insert(
|
||||
"translation_memory_path".to_string(),
|
||||
serde_json::json!(path),
|
||||
);
|
||||
}
|
||||
match options.command {
|
||||
CliCommand::TranslationMemorySummary => {}
|
||||
CliCommand::TranslationMemoryQuery => {
|
||||
let source_text = options
|
||||
.translation_memory_source_text
|
||||
.as_deref()
|
||||
.ok_or_else(|| anyhow::anyhow!("TM query 必须指定 --tm-source-text"))?;
|
||||
let context = parse_translation_memory_context(
|
||||
options.translation_memory_context_json.as_deref(),
|
||||
)?;
|
||||
params.insert("source_text".to_string(), serde_json::json!(source_text));
|
||||
params.insert("source_context".to_string(), serde_json::json!(context));
|
||||
params.insert("limit".to_string(), serde_json::json!(options.query_limit));
|
||||
}
|
||||
CliCommand::TranslationMemoryConfirm => {
|
||||
let record_id = options
|
||||
.translation_memory_record_id
|
||||
.as_deref()
|
||||
.ok_or_else(|| anyhow::anyhow!("TM confirm 必须指定 --tm-record-id"))?;
|
||||
let reviewer = options
|
||||
.translation_memory_reviewer
|
||||
.as_deref()
|
||||
.ok_or_else(|| anyhow::anyhow!("TM confirm 必须指定 --tm-reviewer"))?;
|
||||
params.insert("record_id".to_string(), serde_json::json!(record_id));
|
||||
params.insert("reviewer".to_string(), serde_json::json!(reviewer));
|
||||
if let Some(reason) = options.translation_memory_reason.as_deref() {
|
||||
params.insert("reason".to_string(), serde_json::json!(reason));
|
||||
}
|
||||
}
|
||||
_ => unreachable!(),
|
||||
}
|
||||
Ok(Some(serde_json::Value::Object(params)))
|
||||
}
|
||||
|
||||
pub(super) fn build_translation_memory_summary_report(
|
||||
path: &std::path::Path,
|
||||
) -> anyhow::Result<serde_json::Value> {
|
||||
if !sqlite_file_exists_no_symlink(path, "Translation Memory 数据库")? {
|
||||
return Ok(serde_json::json!({
|
||||
"available": false,
|
||||
"path": path,
|
||||
"reason": "database_missing",
|
||||
}));
|
||||
}
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let summary = runtime.block_on(async {
|
||||
let repository = bat_infrastructure::SqliteTranslationMemoryRepository::open(path)
|
||||
.await
|
||||
.map_err(|error| anyhow::anyhow!("{error}"))?;
|
||||
repository
|
||||
.summary()
|
||||
.await
|
||||
.map_err(|error| anyhow::anyhow!("{error}"))
|
||||
})?;
|
||||
Ok(serde_json::json!({
|
||||
"available": true,
|
||||
"path": path,
|
||||
"schema_version": summary.schema_version,
|
||||
"summary": summary,
|
||||
}))
|
||||
}
|
||||
|
||||
pub(super) fn build_translation_memory_query_report(
|
||||
path: &std::path::Path,
|
||||
source_text: &str,
|
||||
source_context: &TranslationMemoryContext,
|
||||
limit: usize,
|
||||
) -> anyhow::Result<serde_json::Value> {
|
||||
if source_text.trim().is_empty() {
|
||||
return Err(anyhow::anyhow!("TM query 的 source_text 不能为空"));
|
||||
}
|
||||
if !(1..=1000).contains(&limit) {
|
||||
return Err(anyhow::anyhow!("TM query 的 limit 必须在 1..=1000 范围内"));
|
||||
}
|
||||
if !sqlite_file_exists_no_symlink(path, "Translation Memory 数据库")? {
|
||||
return Ok(serde_json::json!({
|
||||
"available": false,
|
||||
"path": path,
|
||||
"source_text": source_text,
|
||||
"source_context": source_context,
|
||||
"matches": [],
|
||||
"reason": "database_missing",
|
||||
}));
|
||||
}
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let matches = runtime.block_on(async {
|
||||
let repository = bat_infrastructure::SqliteTranslationMemoryRepository::open(path)
|
||||
.await
|
||||
.map_err(|error| anyhow::anyhow!("{error}"))?;
|
||||
repository
|
||||
.find_matches(source_text, source_context, limit)
|
||||
.await
|
||||
.map_err(|error| anyhow::anyhow!("{error}"))
|
||||
})?;
|
||||
Ok(serde_json::json!({
|
||||
"available": true,
|
||||
"path": path,
|
||||
"source_text": source_text,
|
||||
"source_context": source_context,
|
||||
"matches": matches,
|
||||
}))
|
||||
}
|
||||
|
||||
pub(super) fn build_translation_memory_confirm_report(
|
||||
path: &std::path::Path,
|
||||
record_id: &str,
|
||||
reviewer: &str,
|
||||
reason: Option<String>,
|
||||
) -> anyhow::Result<serde_json::Value> {
|
||||
if record_id.trim().is_empty() || reviewer.trim().is_empty() {
|
||||
return Err(anyhow::anyhow!(
|
||||
"TM confirm 必须指定非空 record_id 和 reviewer"
|
||||
));
|
||||
}
|
||||
if !sqlite_file_exists_no_symlink(path, "Translation Memory 数据库")? {
|
||||
return Err(anyhow::anyhow!(
|
||||
"Translation Memory 数据库不存在：{}",
|
||||
path.display()
|
||||
));
|
||||
}
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let entry = runtime.block_on(async {
|
||||
let repository = bat_infrastructure::SqliteTranslationMemoryRepository::open(path)
|
||||
.await
|
||||
.map_err(|error| anyhow::anyhow!("{error}"))?;
|
||||
repository
|
||||
.confirm(record_id, reviewer, reason)
|
||||
.await
|
||||
.map_err(|error| anyhow::anyhow!("{error}"))
|
||||
})?;
|
||||
Ok(serde_json::json!({
|
||||
"available": true,
|
||||
"path": path,
|
||||
"entry": entry,
|
||||
}))
|
||||
}
|
||||
|
||||
fn parse_translation_memory_context(
|
||||
value: Option<&str>,
|
||||
) -> anyhow::Result<TranslationMemoryContext> {
|
||||
let Some(value) = value else {
|
||||
return Ok(BTreeMap::new());
|
||||
};
|
||||
serde_json::from_str::<TranslationMemoryContext>(value)
|
||||
.map_err(|error| anyhow::anyhow!("--tm-context-json 必须是 JSON object：{error}"))
|
||||
}
|
||||
|
||||
@@ -261,26 +261,10 @@ pub(super) fn run_translation_worker(options: &CliOptions) -> anyhow::Result<()>
|
||||
.map(|path| lexical_absolute(&path).map_err(anyhow::Error::msg))
|
||||
.transpose()?
|
||||
.unwrap_or(active_official_resource_root(&options.config.output_root)?);
|
||||
let provider = options
|
||||
.translation_provider
|
||||
.as_deref()
|
||||
.unwrap_or(TranslationProviderKind::Mock.as_str());
|
||||
let provider = TranslationProviderKind::parse(provider)
|
||||
.ok_or_else(|| anyhow::anyhow!("i18n worker run 的 provider 无效：{provider}"))?;
|
||||
let config = TranslationWorkerConfig {
|
||||
provider,
|
||||
fixture_path: options.translation_fixture.clone(),
|
||||
concurrency: options.worker_concurrency,
|
||||
max_attempts: options.worker_max_attempts,
|
||||
lease_seconds: options.worker_lease_seconds,
|
||||
retry_backoff: options.worker_retry_backoff,
|
||||
max_tasks: options.worker_max_tasks,
|
||||
worker_id: options
|
||||
.worker_id
|
||||
.clone()
|
||||
.unwrap_or_else(|| format!("bat-worker-{}", std::process::id())),
|
||||
};
|
||||
config.validate()?;
|
||||
let config = super::translation_worker_config_from_options(
|
||||
options,
|
||||
&format!("bat-worker-{}", std::process::id()),
|
||||
)?;
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
|
||||
+18
-12
@@ -29,6 +29,7 @@ pub mod patch_ops;
|
||||
pub mod path_security;
|
||||
pub mod release_flow;
|
||||
pub mod resources;
|
||||
pub mod translation_memory;
|
||||
pub mod translation_tasks;
|
||||
pub mod translation_worker;
|
||||
pub mod translation_workflow;
|
||||
@@ -140,24 +141,29 @@ pub use path_security::{
|
||||
};
|
||||
pub use release_flow::ReleaseFlowStatusCode;
|
||||
pub use resources::{InMemoryResourceRepository, SqliteResourceRepository};
|
||||
pub use translation_memory::{
|
||||
translation_memory_context, translation_memory_repository_path,
|
||||
SqliteTranslationMemoryRepository, TRANSLATION_MEMORY_REPOSITORY_FILE,
|
||||
TRANSLATION_MEMORY_SCHEMA_COMPONENT, TRANSLATION_MEMORY_SCHEMA_VERSION,
|
||||
};
|
||||
pub use translation_tasks::{
|
||||
build_translation_handoff, read_translation_handoff_at, sync_translation_task_repository_at,
|
||||
write_translation_handoff_at, PersistedTranslationTask, PersistedTranslationTaskState,
|
||||
ProviderRun, ProviderRunStatus, SqliteTranslationTaskRepository, TranslationHandoff,
|
||||
TranslationJob, TranslationJobStatus, TranslationTaskFailure, TranslationTaskStatus,
|
||||
TranslationTaskSyncReport, TranslationTaskUnitResult, TranslationUnit, TranslationUnitStatus,
|
||||
TRANSLATION_HANDOFF_FILE, TRANSLATION_HANDOFF_SCHEMA_VERSION, TRANSLATION_TASK_REPOSITORY_FILE,
|
||||
TRANSLATION_TASK_SCHEMA_VERSION,
|
||||
TranslationJob, TranslationJobStatus, TranslationTaskFailure, TranslationTaskResultSourceKind,
|
||||
TranslationTaskStatus, TranslationTaskSyncReport, TranslationTaskUnitResult, TranslationUnit,
|
||||
TranslationUnitStatus, TRANSLATION_HANDOFF_FILE, TRANSLATION_HANDOFF_SCHEMA_VERSION,
|
||||
TRANSLATION_TASK_REPOSITORY_FILE, TRANSLATION_TASK_SCHEMA_VERSION,
|
||||
};
|
||||
pub use translation_worker::{
|
||||
run_translation_worker_at, run_translation_worker_with_provider, CrowdinProvider,
|
||||
MockTranslationProvider, TranslationProvider, TranslationProviderFailureClass,
|
||||
TranslationProviderKind, TranslationProviderRequest, TranslationProviderResponse,
|
||||
TranslationProviderUnit, TranslationProviderUnitResult, TranslationWorkerConfig,
|
||||
TranslationWorkerFailure, TranslationWorkerReport, DEFAULT_TRANSLATION_CONCURRENCY,
|
||||
DEFAULT_TRANSLATION_LEASE_SECONDS, DEFAULT_TRANSLATION_MAX_ATTEMPTS,
|
||||
DEFAULT_TRANSLATION_RETRY_BACKOFF, MAX_TRANSLATION_CONCURRENCY, MIN_TRANSLATION_CONCURRENCY,
|
||||
MOCK_TRANSLATION_FIXTURE_VERSION,
|
||||
run_translation_worker_at, run_translation_worker_at_with_cancellation,
|
||||
run_translation_worker_with_provider, CrowdinProvider, MockTranslationProvider,
|
||||
TranslationProvider, TranslationProviderFailureClass, TranslationProviderKind,
|
||||
TranslationProviderRequest, TranslationProviderResponse, TranslationProviderUnit,
|
||||
TranslationProviderUnitResult, TranslationWorkerConfig, TranslationWorkerFailure,
|
||||
TranslationWorkerReport, DEFAULT_TRANSLATION_CONCURRENCY, DEFAULT_TRANSLATION_LEASE_SECONDS,
|
||||
DEFAULT_TRANSLATION_MAX_ATTEMPTS, DEFAULT_TRANSLATION_RETRY_BACKOFF,
|
||||
MAX_TRANSLATION_CONCURRENCY, MIN_TRANSLATION_CONCURRENCY, MOCK_TRANSLATION_FIXTURE_VERSION,
|
||||
};
|
||||
pub use translation_workflow::{
|
||||
completed_worker_translation_workbench, export_completed_worker_translation_workbench,
|
||||
|
||||
@@ -91,6 +91,12 @@ pub struct LocalizedPatchOperationMetadata {
|
||||
/// Provider run ID that produced the text, if applicable.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub provider_run_id: Option<String>,
|
||||
/// Source kind of the translation result.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub translation_source_kind: Option<String>,
|
||||
/// Trusted Translation Memory record used for the text, if applicable.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub translation_memory_record_id: Option<String>,
|
||||
/// Review state used by the publication input.
|
||||
pub review_status: String,
|
||||
}
|
||||
@@ -331,6 +337,12 @@ pub struct LocalizedPatchOperation {
|
||||
/// Provider run ID that produced the text, if applicable.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub provider_run_id: Option<String>,
|
||||
/// Source kind of the translation result.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub translation_source_kind: Option<String>,
|
||||
/// Trusted Translation Memory record used for the text, if applicable.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub translation_memory_record_id: Option<String>,
|
||||
/// Review state used by the publication input.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub review_status: Option<String>,
|
||||
@@ -941,6 +953,8 @@ impl LocalizedPatchOperation {
|
||||
source_text_blake3: None,
|
||||
translation_provider: None,
|
||||
provider_run_id: None,
|
||||
translation_source_kind: None,
|
||||
translation_memory_record_id: None,
|
||||
review_status: None,
|
||||
},
|
||||
metadata,
|
||||
@@ -966,6 +980,8 @@ impl LocalizedPatchOperation {
|
||||
source_text_blake3: None,
|
||||
translation_provider: None,
|
||||
provider_run_id: None,
|
||||
translation_source_kind: None,
|
||||
translation_memory_record_id: None,
|
||||
review_status: None,
|
||||
},
|
||||
metadata,
|
||||
@@ -990,6 +1006,8 @@ impl LocalizedPatchOperation {
|
||||
source_text_blake3: None,
|
||||
translation_provider: None,
|
||||
provider_run_id: None,
|
||||
translation_source_kind: None,
|
||||
translation_memory_record_id: None,
|
||||
review_status: None,
|
||||
},
|
||||
metadata,
|
||||
@@ -1005,6 +1023,8 @@ impl LocalizedPatchOperation {
|
||||
operation.source_text_blake3 = Some(metadata.source_text_blake3.clone());
|
||||
operation.translation_provider = metadata.translation_provider.clone();
|
||||
operation.provider_run_id = metadata.provider_run_id.clone();
|
||||
operation.translation_source_kind = metadata.translation_source_kind.clone();
|
||||
operation.translation_memory_record_id = metadata.translation_memory_record_id.clone();
|
||||
operation.review_status = Some(metadata.review_status.clone());
|
||||
}
|
||||
operation
|
||||
@@ -1029,6 +1049,8 @@ impl Default for LocalizedPatchOperation {
|
||||
source_text_blake3: None,
|
||||
translation_provider: None,
|
||||
provider_run_id: None,
|
||||
translation_source_kind: None,
|
||||
translation_memory_record_id: None,
|
||||
review_status: None,
|
||||
}
|
||||
}
|
||||
@@ -1401,6 +1423,8 @@ mod tests {
|
||||
source_text_blake3: Some(blake3::hash(source).to_hex().to_string()),
|
||||
translation_provider: Some("mock".to_string()),
|
||||
provider_run_id: Some("mock:unit-1:attempt-1".to_string()),
|
||||
translation_source_kind: Some("provider".to_string()),
|
||||
translation_memory_record_id: None,
|
||||
review_status: Some("provider_completed".to_string()),
|
||||
}],
|
||||
}],
|
||||
|
||||
@@ -0,0 +1,859 @@
|
||||
//! 跨 official release 的 Translation Memory SQLite 仓储。
|
||||
|
||||
use crate::path_security::{set_file_mode, STATE_FILE_MODE};
|
||||
use async_trait::async_trait;
|
||||
use bat_core::domain::{
|
||||
TranslationMemoryContext, TranslationMemoryDraft, TranslationMemoryEntry,
|
||||
TranslationMemoryMatch, TranslationMemoryMatchKind, TranslationMemorySourceKind,
|
||||
TranslationMemorySummary, TranslationMemoryTrustStatus,
|
||||
};
|
||||
use bat_core::repositories::TranslationMemoryRepository;
|
||||
use bat_core::{Error, Result};
|
||||
use serde::de::DeserializeOwned;
|
||||
use sqlx::sqlite::{SqliteConnectOptions, SqliteJournalMode, SqlitePoolOptions};
|
||||
use sqlx::{Row, SqlitePool};
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::str::FromStr;
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
|
||||
/// TM SQLite schema 版本。
|
||||
pub const TRANSLATION_MEMORY_SCHEMA_VERSION: u32 = 1;
|
||||
/// TM schema migration component。
|
||||
pub const TRANSLATION_MEMORY_SCHEMA_COMPONENT: &str = "translation_memory";
|
||||
/// 默认 TM 数据库文件名。
|
||||
pub const TRANSLATION_MEMORY_REPOSITORY_FILE: &str = "translation-memory.sqlite";
|
||||
|
||||
/// SQLite-backed Translation Memory 仓储。
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct SqliteTranslationMemoryRepository {
|
||||
pub(crate) pool: SqlitePool,
|
||||
}
|
||||
|
||||
impl SqliteTranslationMemoryRepository {
|
||||
/// 创建或打开 TM 数据库并执行迁移。
|
||||
pub async fn new(path: impl AsRef<Path>) -> Result<Self> {
|
||||
Self::open_with(path.as_ref(), true).await
|
||||
}
|
||||
|
||||
/// 只打开已有 TM 数据库，不创建新文件。
|
||||
pub async fn open(path: impl AsRef<Path>) -> Result<Self> {
|
||||
Self::open_with(path.as_ref(), false).await
|
||||
}
|
||||
|
||||
/// 根据 active release 根目录计算默认的跨 release TM 路径。
|
||||
///
|
||||
/// 正式 release 根目录形如 `<output>/versions/<id>`，因此默认结果为
|
||||
/// `<output>/translation-memory.sqlite`，不会写入已发布版本目录。
|
||||
pub fn repository_path(resource_root: &Path) -> PathBuf {
|
||||
if resource_root
|
||||
.parent()
|
||||
.and_then(Path::file_name)
|
||||
.is_some_and(|name| name == "versions")
|
||||
{
|
||||
if let Some(output_root) = resource_root.parent().and_then(Path::parent) {
|
||||
return output_root.join(TRANSLATION_MEMORY_REPOSITORY_FILE);
|
||||
}
|
||||
}
|
||||
resource_root.join(TRANSLATION_MEMORY_REPOSITORY_FILE)
|
||||
}
|
||||
|
||||
async fn open_with(path: &Path, create_if_missing: bool) -> Result<Self> {
|
||||
let absolute = bat_infrastructure_absolute(path)?;
|
||||
let parent = absolute.parent().ok_or_else(|| {
|
||||
Error::InvalidArgument(format!("TM 数据库缺少父目录：{}", absolute.display()))
|
||||
})?;
|
||||
ensure_safe_tm_parent(parent)?;
|
||||
if create_if_missing {
|
||||
tokio::fs::create_dir_all(parent).await?;
|
||||
ensure_safe_tm_parent(parent)?;
|
||||
}
|
||||
if let Ok(metadata) = fs::symlink_metadata(&absolute) {
|
||||
if metadata.file_type().is_symlink() {
|
||||
return Err(Error::InvalidArgument(format!(
|
||||
"TM 数据库不能是 symlink：{}",
|
||||
absolute.display()
|
||||
)));
|
||||
}
|
||||
if !metadata.is_file() {
|
||||
return Err(Error::InvalidArgument(format!(
|
||||
"TM 数据库不是普通文件：{}",
|
||||
absolute.display()
|
||||
)));
|
||||
}
|
||||
} else if !create_if_missing {
|
||||
return Err(Error::NotFound(absolute.display().to_string()));
|
||||
}
|
||||
|
||||
let options = SqliteConnectOptions::from_str(&format!("sqlite://{}", absolute.display()))
|
||||
.map_err(|error| Error::Other(error.into()))?
|
||||
.create_if_missing(create_if_missing)
|
||||
.journal_mode(SqliteJournalMode::Wal)
|
||||
.busy_timeout(Duration::from_secs(30));
|
||||
let pool = SqlitePoolOptions::new()
|
||||
.max_connections(1)
|
||||
.connect_with(options)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
set_file_mode(&absolute, STATE_FILE_MODE, "Translation Memory 数据库")
|
||||
.map_err(Error::InvalidArgument)?;
|
||||
let repository = Self { pool };
|
||||
repository.init_schema().await?;
|
||||
Ok(repository)
|
||||
}
|
||||
|
||||
async fn init_schema(&self) -> Result<()> {
|
||||
sqlx::query(
|
||||
r#"
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
component TEXT PRIMARY KEY NOT NULL,
|
||||
version INTEGER NOT NULL CHECK(version >= 1)
|
||||
)
|
||||
"#,
|
||||
)
|
||||
.execute(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
sqlx::query(
|
||||
r#"
|
||||
CREATE TABLE IF NOT EXISTS translation_memory (
|
||||
record_id TEXT PRIMARY KEY NOT NULL,
|
||||
source_text TEXT NOT NULL,
|
||||
source_hash TEXT NOT NULL,
|
||||
normalized_source_text TEXT NOT NULL,
|
||||
source_context_json TEXT NOT NULL,
|
||||
source_context_hash TEXT NOT NULL,
|
||||
translated_text TEXT NOT NULL,
|
||||
translation_source_kind TEXT NOT NULL,
|
||||
trust_status TEXT NOT NULL,
|
||||
official_release_id TEXT NOT NULL,
|
||||
source_trace_json TEXT NOT NULL,
|
||||
provider TEXT,
|
||||
provider_run_id TEXT,
|
||||
created_unix_seconds INTEGER NOT NULL,
|
||||
updated_unix_seconds INTEGER NOT NULL,
|
||||
trusted_unix_seconds INTEGER,
|
||||
trusted_by TEXT,
|
||||
trusted_reason TEXT,
|
||||
supersedes_record_id TEXT,
|
||||
superseded_by_record_id TEXT,
|
||||
CHECK (length(source_text) > 0),
|
||||
CHECK (length(source_hash) > 0),
|
||||
CHECK (length(source_context_hash) > 0),
|
||||
CHECK (length(official_release_id) > 0),
|
||||
CHECK (translation_source_kind IN ('provider', 'manual', 'imported')),
|
||||
CHECK (trust_status IN ('candidate', 'trusted', 'superseded', 'rejected'))
|
||||
)
|
||||
"#,
|
||||
)
|
||||
.execute(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
sqlx::query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_translation_memory_source_hash \
|
||||
ON translation_memory(source_hash)",
|
||||
)
|
||||
.execute(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
sqlx::query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_translation_memory_normalized_source \
|
||||
ON translation_memory(normalized_source_text)",
|
||||
)
|
||||
.execute(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
sqlx::query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_translation_memory_context \
|
||||
ON translation_memory(source_hash, source_context_hash)",
|
||||
)
|
||||
.execute(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
|
||||
let current: Option<i64> =
|
||||
sqlx::query_scalar("SELECT version FROM schema_migrations WHERE component = ?1")
|
||||
.bind(TRANSLATION_MEMORY_SCHEMA_COMPONENT)
|
||||
.fetch_optional(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
if current.is_some_and(|version| version > i64::from(TRANSLATION_MEMORY_SCHEMA_VERSION)) {
|
||||
return Err(Error::InvalidArgument(format!(
|
||||
"不支持的 Translation Memory schema 版本：{}",
|
||||
current.unwrap_or_default()
|
||||
)));
|
||||
}
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO schema_migrations(component, version)
|
||||
VALUES (?1, ?2)
|
||||
ON CONFLICT(component) DO UPDATE SET version = excluded.version
|
||||
"#,
|
||||
)
|
||||
.bind(TRANSLATION_MEMORY_SCHEMA_COMPONENT)
|
||||
.bind(i64::from(TRANSLATION_MEMORY_SCHEMA_VERSION))
|
||||
.execute(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn find_optional(&self, record_id: &str) -> Result<Option<TranslationMemoryEntry>> {
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
SELECT record_id, source_text, source_hash, normalized_source_text,
|
||||
source_context_json, source_context_hash, translated_text,
|
||||
translation_source_kind, trust_status, official_release_id,
|
||||
source_trace_json, provider, provider_run_id,
|
||||
created_unix_seconds, updated_unix_seconds, trusted_unix_seconds,
|
||||
trusted_by, trusted_reason, supersedes_record_id, superseded_by_record_id
|
||||
FROM translation_memory
|
||||
WHERE record_id = ?1
|
||||
"#,
|
||||
)
|
||||
.bind(record_id)
|
||||
.fetch_optional(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
row.map(row_to_entry).transpose()
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl TranslationMemoryRepository for SqliteTranslationMemoryRepository {
|
||||
async fn upsert_candidate(
|
||||
&self,
|
||||
draft: TranslationMemoryDraft,
|
||||
) -> Result<TranslationMemoryEntry> {
|
||||
validate_draft(&draft)?;
|
||||
let entry = entry_from_draft(draft)?;
|
||||
if let Some(existing) = self.find_optional(&entry.record_id).await? {
|
||||
if existing.trust_status != TranslationMemoryTrustStatus::Candidate {
|
||||
return Ok(existing);
|
||||
}
|
||||
let source_trace_json = serde_json::to_string(&entry.source_trace)
|
||||
.map_err(|error| Error::Serialization(error.to_string()))?;
|
||||
sqlx::query(
|
||||
r#"
|
||||
UPDATE translation_memory
|
||||
SET source_trace_json = ?2, provider = ?3, provider_run_id = ?4,
|
||||
updated_unix_seconds = ?5
|
||||
WHERE record_id = ?1 AND trust_status = 'candidate'
|
||||
"#,
|
||||
)
|
||||
.bind(&entry.record_id)
|
||||
.bind(source_trace_json)
|
||||
.bind(&entry.provider)
|
||||
.bind(&entry.provider_run_id)
|
||||
.bind(i64::try_from(entry.updated_unix_seconds).unwrap_or(i64::MAX))
|
||||
.execute(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
return self.find(&entry.record_id).await;
|
||||
}
|
||||
|
||||
let source_context_json = serde_json::to_string(&entry.source_context)
|
||||
.map_err(|error| Error::Serialization(error.to_string()))?;
|
||||
let source_trace_json = serde_json::to_string(&entry.source_trace)
|
||||
.map_err(|error| Error::Serialization(error.to_string()))?;
|
||||
let result = sqlx::query(
|
||||
r#"
|
||||
INSERT INTO translation_memory (
|
||||
record_id, source_text, source_hash, normalized_source_text,
|
||||
source_context_json, source_context_hash, translated_text,
|
||||
translation_source_kind, trust_status, official_release_id,
|
||||
source_trace_json, provider, provider_run_id,
|
||||
created_unix_seconds, updated_unix_seconds,
|
||||
trusted_unix_seconds, trusted_by, trusted_reason,
|
||||
supersedes_record_id, superseded_by_record_id
|
||||
)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10,
|
||||
?11, ?12, ?13, ?14, ?14, ?15, ?16, ?17, ?18, ?19)
|
||||
ON CONFLICT(record_id) DO UPDATE SET
|
||||
source_trace_json = excluded.source_trace_json,
|
||||
provider = excluded.provider,
|
||||
provider_run_id = excluded.provider_run_id,
|
||||
updated_unix_seconds = excluded.updated_unix_seconds
|
||||
WHERE translation_memory.trust_status = 'candidate'
|
||||
"#,
|
||||
)
|
||||
.bind(&entry.record_id)
|
||||
.bind(&entry.source_text)
|
||||
.bind(&entry.source_hash)
|
||||
.bind(&entry.normalized_source_text)
|
||||
.bind(source_context_json)
|
||||
.bind(&entry.source_context_hash)
|
||||
.bind(&entry.translated_text)
|
||||
.bind(entry.translation_source_kind.as_str())
|
||||
.bind(entry.trust_status.as_str())
|
||||
.bind(&entry.official_release_id)
|
||||
.bind(source_trace_json)
|
||||
.bind(&entry.provider)
|
||||
.bind(&entry.provider_run_id)
|
||||
.bind(i64::try_from(entry.created_unix_seconds).unwrap_or(i64::MAX))
|
||||
.bind(
|
||||
entry
|
||||
.trusted_unix_seconds
|
||||
.map(|value| i64::try_from(value).unwrap_or(i64::MAX)),
|
||||
)
|
||||
.bind(&entry.trusted_by)
|
||||
.bind(&entry.trusted_reason)
|
||||
.bind(&entry.supersedes_record_id)
|
||||
.bind(&entry.superseded_by_record_id)
|
||||
.execute(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
if result.rows_affected() == 0 {
|
||||
return self.find(&entry.record_id).await;
|
||||
}
|
||||
Ok(entry)
|
||||
}
|
||||
|
||||
async fn find_matches(
|
||||
&self,
|
||||
source_text: &str,
|
||||
source_context: &TranslationMemoryContext,
|
||||
limit: usize,
|
||||
) -> Result<Vec<TranslationMemoryMatch>> {
|
||||
if source_text.is_empty() {
|
||||
return Err(Error::InvalidArgument(
|
||||
"Translation Memory 查询 source_text 不能为空".to_string(),
|
||||
));
|
||||
}
|
||||
if limit == 0 {
|
||||
return Err(Error::InvalidArgument(
|
||||
"Translation Memory 查询 limit 必须大于 0".to_string(),
|
||||
));
|
||||
}
|
||||
let source_hash = hash_text(source_text);
|
||||
let normalized_source_text = normalize_source_text(source_text);
|
||||
let rows = sqlx::query(
|
||||
r#"
|
||||
SELECT record_id, source_text, source_hash, normalized_source_text,
|
||||
source_context_json, source_context_hash, translated_text,
|
||||
translation_source_kind, trust_status, official_release_id,
|
||||
source_trace_json, provider, provider_run_id,
|
||||
created_unix_seconds, updated_unix_seconds, trusted_unix_seconds,
|
||||
trusted_by, trusted_reason, supersedes_record_id, superseded_by_record_id
|
||||
FROM translation_memory
|
||||
WHERE source_hash = ?1 OR normalized_source_text = ?2
|
||||
ORDER BY updated_unix_seconds DESC, record_id ASC
|
||||
"#,
|
||||
)
|
||||
.bind(source_hash)
|
||||
.bind(&normalized_source_text)
|
||||
.fetch_all(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
let mut matches = rows
|
||||
.into_iter()
|
||||
.map(row_to_entry)
|
||||
.collect::<Result<Vec<_>>>()?
|
||||
.into_iter()
|
||||
.filter_map(|entry| {
|
||||
let raw_exact = entry.source_text == source_text;
|
||||
let normalized_exact = entry.normalized_source_text == normalized_source_text;
|
||||
if !raw_exact && !normalized_exact {
|
||||
return None;
|
||||
}
|
||||
let same_context = !source_context.is_empty()
|
||||
&& !entry.source_context.is_empty()
|
||||
&& entry.source_context == *source_context;
|
||||
let strong = raw_exact
|
||||
&& same_context
|
||||
&& entry.trust_status == TranslationMemoryTrustStatus::Trusted;
|
||||
let match_kind = if strong {
|
||||
TranslationMemoryMatchKind::StrongExact
|
||||
} else if raw_exact {
|
||||
TranslationMemoryMatchKind::CandidateExact
|
||||
} else {
|
||||
TranslationMemoryMatchKind::SourceOnly
|
||||
};
|
||||
Some(TranslationMemoryMatch {
|
||||
can_auto_reuse: strong,
|
||||
entry,
|
||||
match_kind,
|
||||
})
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
matches.sort_by(|left, right| {
|
||||
match_rank(left)
|
||||
.cmp(&match_rank(right))
|
||||
.then_with(|| {
|
||||
right
|
||||
.entry
|
||||
.updated_unix_seconds
|
||||
.cmp(&left.entry.updated_unix_seconds)
|
||||
})
|
||||
.then_with(|| left.entry.record_id.cmp(&right.entry.record_id))
|
||||
});
|
||||
matches.truncate(limit);
|
||||
Ok(matches)
|
||||
}
|
||||
|
||||
async fn confirm(
|
||||
&self,
|
||||
record_id: &str,
|
||||
reviewer: &str,
|
||||
reason: Option<String>,
|
||||
) -> Result<TranslationMemoryEntry> {
|
||||
if reviewer.trim().is_empty() {
|
||||
return Err(Error::InvalidArgument(
|
||||
"Translation Memory reviewer 不能为空".to_string(),
|
||||
));
|
||||
}
|
||||
let current = self.find(record_id).await?;
|
||||
if current.trust_status == TranslationMemoryTrustStatus::Trusted {
|
||||
return Ok(current);
|
||||
}
|
||||
if current.trust_status != TranslationMemoryTrustStatus::Candidate {
|
||||
return Err(Error::InvalidArgument(format!(
|
||||
"Translation Memory 记录 {} 当前状态为 {}，不能确认",
|
||||
record_id,
|
||||
current.trust_status.as_str()
|
||||
)));
|
||||
}
|
||||
let now = unix_seconds_now();
|
||||
sqlx::query(
|
||||
r#"
|
||||
UPDATE translation_memory
|
||||
SET trust_status = 'trusted', trusted_unix_seconds = ?2,
|
||||
trusted_by = ?3, trusted_reason = ?4, updated_unix_seconds = ?2
|
||||
WHERE record_id = ?1 AND trust_status = 'candidate'
|
||||
"#,
|
||||
)
|
||||
.bind(record_id)
|
||||
.bind(i64::try_from(now).unwrap_or(i64::MAX))
|
||||
.bind(reviewer.trim())
|
||||
.bind(reason.filter(|value| !value.trim().is_empty()))
|
||||
.execute(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
self.find(record_id).await
|
||||
}
|
||||
|
||||
async fn find(&self, record_id: &str) -> Result<TranslationMemoryEntry> {
|
||||
self.find_optional(record_id)
|
||||
.await?
|
||||
.ok_or_else(|| Error::NotFound(record_id.to_string()))
|
||||
}
|
||||
|
||||
async fn summary(&self) -> Result<TranslationMemorySummary> {
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
SELECT COUNT(*) AS record_count,
|
||||
SUM(CASE WHEN trust_status = 'trusted' THEN 1 ELSE 0 END) AS trusted_count,
|
||||
SUM(CASE WHEN trust_status = 'candidate' THEN 1 ELSE 0 END) AS candidate_count,
|
||||
SUM(CASE WHEN trust_status = 'superseded' THEN 1 ELSE 0 END) AS superseded_count,
|
||||
SUM(CASE WHEN trust_status = 'rejected' THEN 1 ELSE 0 END) AS rejected_count
|
||||
FROM translation_memory
|
||||
"#,
|
||||
)
|
||||
.fetch_one(&self.pool)
|
||||
.await
|
||||
.map_err(db_error)?;
|
||||
Ok(TranslationMemorySummary {
|
||||
schema_version: TRANSLATION_MEMORY_SCHEMA_VERSION,
|
||||
record_count: row.try_get::<i64, _>("record_count").map_err(db_error)? as u64,
|
||||
trusted_count: row.try_get::<i64, _>("trusted_count").map_err(db_error)? as u64,
|
||||
candidate_count: row.try_get::<i64, _>("candidate_count").map_err(db_error)? as u64,
|
||||
superseded_count: row
|
||||
.try_get::<i64, _>("superseded_count")
|
||||
.map_err(db_error)? as u64,
|
||||
rejected_count: row.try_get::<i64, _>("rejected_count").map_err(db_error)? as u64,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// 由 active official release 根目录计算默认 TM 数据库路径。
|
||||
pub fn translation_memory_repository_path(resource_root: &Path) -> PathBuf {
|
||||
SqliteTranslationMemoryRepository::repository_path(resource_root)
|
||||
}
|
||||
|
||||
/// 从 TextUnit 定位字段构建 TM 上下文。
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn translation_memory_context(
|
||||
destination: &str,
|
||||
archive_entry: Option<&str>,
|
||||
serialized_file: Option<&str>,
|
||||
path_id: Option<i64>,
|
||||
class_id: Option<i32>,
|
||||
field_path: Option<&str>,
|
||||
format: Option<&str>,
|
||||
asset_name: Option<&str>,
|
||||
text_source_kind: Option<&str>,
|
||||
parser_context: &TranslationMemoryContext,
|
||||
) -> TranslationMemoryContext {
|
||||
let mut context = TranslationMemoryContext::new();
|
||||
context.insert("destination".to_string(), destination.to_string());
|
||||
insert_optional(&mut context, "archive_entry", archive_entry);
|
||||
insert_optional(&mut context, "serialized_file", serialized_file);
|
||||
if let Some(value) = path_id {
|
||||
context.insert("path_id".to_string(), value.to_string());
|
||||
}
|
||||
if let Some(value) = class_id {
|
||||
context.insert("class_id".to_string(), value.to_string());
|
||||
}
|
||||
insert_optional(&mut context, "field_path", field_path);
|
||||
insert_optional(&mut context, "format", format);
|
||||
insert_optional(&mut context, "asset_name", asset_name);
|
||||
insert_optional(&mut context, "text_source_kind", text_source_kind);
|
||||
for (key, value) in parser_context {
|
||||
context.insert(format!("context.{key}"), value.clone());
|
||||
}
|
||||
context
|
||||
}
|
||||
|
||||
fn insert_optional(context: &mut TranslationMemoryContext, key: &str, value: Option<&str>) {
|
||||
if let Some(value) = value.filter(|value| !value.is_empty()) {
|
||||
context.insert(key.to_string(), value.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
fn entry_from_draft(draft: TranslationMemoryDraft) -> Result<TranslationMemoryEntry> {
|
||||
let source_hash = hash_text(&draft.source_text);
|
||||
let normalized_source_text = normalize_source_text(&draft.source_text);
|
||||
let source_context_hash = hash_context(&draft.source_context)?;
|
||||
let source_kind = draft.translation_source_kind;
|
||||
let record_id = record_id(
|
||||
&source_hash,
|
||||
&source_context_hash,
|
||||
&draft.translated_text,
|
||||
&draft.official_release_id,
|
||||
&source_kind,
|
||||
);
|
||||
let observed = draft.observed_unix_seconds;
|
||||
Ok(TranslationMemoryEntry {
|
||||
record_id,
|
||||
source_text: draft.source_text,
|
||||
source_hash,
|
||||
normalized_source_text,
|
||||
source_context: draft.source_context,
|
||||
source_context_hash,
|
||||
translated_text: draft.translated_text,
|
||||
translation_source_kind: source_kind,
|
||||
trust_status: TranslationMemoryTrustStatus::Candidate,
|
||||
official_release_id: draft.official_release_id,
|
||||
source_trace: draft.source_trace,
|
||||
provider: draft.provider,
|
||||
provider_run_id: draft.provider_run_id,
|
||||
created_unix_seconds: observed,
|
||||
updated_unix_seconds: observed,
|
||||
trusted_unix_seconds: None,
|
||||
trusted_by: None,
|
||||
trusted_reason: None,
|
||||
supersedes_record_id: None,
|
||||
superseded_by_record_id: None,
|
||||
})
|
||||
}
|
||||
|
||||
fn validate_draft(draft: &TranslationMemoryDraft) -> Result<()> {
|
||||
if draft.source_text.is_empty() {
|
||||
return Err(Error::InvalidArgument(
|
||||
"Translation Memory source_text 不能为空".to_string(),
|
||||
));
|
||||
}
|
||||
if draft.translated_text.trim().is_empty() {
|
||||
return Err(Error::InvalidArgument(
|
||||
"Translation Memory translated_text 不能为空".to_string(),
|
||||
));
|
||||
}
|
||||
if draft.official_release_id.trim().is_empty()
|
||||
|| draft.source_trace.official_release_id.trim().is_empty()
|
||||
{
|
||||
return Err(Error::InvalidArgument(
|
||||
"Translation Memory official_release_id 不能为空".to_string(),
|
||||
));
|
||||
}
|
||||
if draft.official_release_id != draft.source_trace.official_release_id {
|
||||
return Err(Error::InvalidArgument(
|
||||
"Translation Memory draft 的 release provenance 不一致".to_string(),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn row_to_entry(row: sqlx::sqlite::SqliteRow) -> Result<TranslationMemoryEntry> {
|
||||
let source_context = parse_json(row.try_get("source_context_json").map_err(db_error)?)?;
|
||||
let source_trace = parse_json(row.try_get("source_trace_json").map_err(db_error)?)?;
|
||||
Ok(TranslationMemoryEntry {
|
||||
record_id: row.try_get("record_id").map_err(db_error)?,
|
||||
source_text: row.try_get("source_text").map_err(db_error)?,
|
||||
source_hash: row.try_get("source_hash").map_err(db_error)?,
|
||||
normalized_source_text: row.try_get("normalized_source_text").map_err(db_error)?,
|
||||
source_context,
|
||||
source_context_hash: row.try_get("source_context_hash").map_err(db_error)?,
|
||||
translated_text: row.try_get("translated_text").map_err(db_error)?,
|
||||
translation_source_kind: parse_source_kind(
|
||||
row.try_get::<String, _>("translation_source_kind")
|
||||
.map_err(db_error)?
|
||||
.as_str(),
|
||||
)?,
|
||||
trust_status: parse_trust_status(
|
||||
row.try_get::<String, _>("trust_status")
|
||||
.map_err(db_error)?
|
||||
.as_str(),
|
||||
)?,
|
||||
official_release_id: row.try_get("official_release_id").map_err(db_error)?,
|
||||
source_trace,
|
||||
provider: row.try_get("provider").map_err(db_error)?,
|
||||
provider_run_id: row.try_get("provider_run_id").map_err(db_error)?,
|
||||
created_unix_seconds: i64_to_u64(
|
||||
row.try_get("created_unix_seconds").map_err(db_error)?,
|
||||
"created",
|
||||
)?,
|
||||
updated_unix_seconds: i64_to_u64(
|
||||
row.try_get("updated_unix_seconds").map_err(db_error)?,
|
||||
"updated",
|
||||
)?,
|
||||
trusted_unix_seconds: optional_i64_to_u64(
|
||||
row.try_get("trusted_unix_seconds").map_err(db_error)?,
|
||||
"trusted",
|
||||
)?,
|
||||
trusted_by: row.try_get("trusted_by").map_err(db_error)?,
|
||||
trusted_reason: row.try_get("trusted_reason").map_err(db_error)?,
|
||||
supersedes_record_id: row.try_get("supersedes_record_id").map_err(db_error)?,
|
||||
superseded_by_record_id: row.try_get("superseded_by_record_id").map_err(db_error)?,
|
||||
})
|
||||
}
|
||||
|
||||
fn parse_json<T: DeserializeOwned>(value: String) -> Result<T> {
|
||||
serde_json::from_str(&value).map_err(|error| Error::Serialization(error.to_string()))
|
||||
}
|
||||
|
||||
fn parse_source_kind(value: &str) -> Result<TranslationMemorySourceKind> {
|
||||
match value {
|
||||
"provider" => Ok(TranslationMemorySourceKind::Provider),
|
||||
"manual" => Ok(TranslationMemorySourceKind::Manual),
|
||||
"imported" => Ok(TranslationMemorySourceKind::Imported),
|
||||
_ => Err(Error::Serialization(format!(
|
||||
"未知 Translation Memory source kind：{value}"
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_trust_status(value: &str) -> Result<TranslationMemoryTrustStatus> {
|
||||
match value {
|
||||
"candidate" => Ok(TranslationMemoryTrustStatus::Candidate),
|
||||
"trusted" => Ok(TranslationMemoryTrustStatus::Trusted),
|
||||
"superseded" => Ok(TranslationMemoryTrustStatus::Superseded),
|
||||
"rejected" => Ok(TranslationMemoryTrustStatus::Rejected),
|
||||
_ => Err(Error::Serialization(format!(
|
||||
"未知 Translation Memory trust status：{value}"
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
fn match_rank(value: &TranslationMemoryMatch) -> u8 {
|
||||
match value.match_kind {
|
||||
TranslationMemoryMatchKind::StrongExact if value.can_auto_reuse => 0,
|
||||
TranslationMemoryMatchKind::CandidateExact => 1,
|
||||
TranslationMemoryMatchKind::SourceOnly => 2,
|
||||
TranslationMemoryMatchKind::StrongExact => 1,
|
||||
}
|
||||
}
|
||||
|
||||
fn hash_text(value: &str) -> String {
|
||||
blake3::hash(value.as_bytes()).to_hex().to_string()
|
||||
}
|
||||
|
||||
fn normalize_source_text(value: &str) -> String {
|
||||
value.replace("\r\n", "\n").replace('\r', "\n")
|
||||
}
|
||||
|
||||
fn hash_context(context: &TranslationMemoryContext) -> Result<String> {
|
||||
let bytes =
|
||||
serde_json::to_vec(context).map_err(|error| Error::Serialization(error.to_string()))?;
|
||||
Ok(blake3::hash(&bytes).to_hex().to_string())
|
||||
}
|
||||
|
||||
fn record_id(
|
||||
source_hash: &str,
|
||||
context_hash: &str,
|
||||
translated_text: &str,
|
||||
official_release_id: &str,
|
||||
source_kind: &TranslationMemorySourceKind,
|
||||
) -> String {
|
||||
let mut key = Vec::new();
|
||||
for value in [
|
||||
source_hash,
|
||||
context_hash,
|
||||
translated_text,
|
||||
official_release_id,
|
||||
source_kind.as_str(),
|
||||
] {
|
||||
key.extend_from_slice(value.as_bytes());
|
||||
key.push(0);
|
||||
}
|
||||
format!("tm-{}", blake3::hash(&key).to_hex())
|
||||
}
|
||||
|
||||
fn unix_seconds_now() -> u64 {
|
||||
SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.unwrap_or_default()
|
||||
.as_secs()
|
||||
}
|
||||
|
||||
fn i64_to_u64(value: i64, label: &str) -> Result<u64> {
|
||||
u64::try_from(value)
|
||||
.map_err(|_| Error::Serialization(format!("Translation Memory {label} 时间无效")))
|
||||
}
|
||||
|
||||
fn optional_i64_to_u64(value: Option<i64>, label: &str) -> Result<Option<u64>> {
|
||||
value.map(|value| i64_to_u64(value, label)).transpose()
|
||||
}
|
||||
|
||||
fn db_error(error: sqlx::Error) -> Error {
|
||||
Error::Other(error.into())
|
||||
}
|
||||
|
||||
fn bat_infrastructure_absolute(path: &Path) -> Result<PathBuf> {
|
||||
crate::path_security::lexical_absolute(path).map_err(Error::InvalidArgument)
|
||||
}
|
||||
|
||||
fn ensure_safe_tm_parent(parent: &Path) -> Result<()> {
|
||||
crate::path_security::ensure_safe_directory_path(parent, "Translation Memory 数据库")
|
||||
.map_err(Error::InvalidArgument)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use bat_core::domain::TranslationMemorySourceTrace;
|
||||
|
||||
fn draft(release: &str, source: &str, translated: &str) -> TranslationMemoryDraft {
|
||||
let source_trace = TranslationMemorySourceTrace {
|
||||
official_release_id: release.to_string(),
|
||||
unit_id: Some(format!("{release}-unit")),
|
||||
task_id: Some(format!("{release}-task")),
|
||||
destination: Some("Bundles/story.bundle".to_string()),
|
||||
archive_entry: None,
|
||||
serialized_file: Some("CAB-story".to_string()),
|
||||
path_id: Some(1),
|
||||
class_id: Some(49),
|
||||
field_path: Some("m_Text".to_string()),
|
||||
format: Some("plain".to_string()),
|
||||
asset_name: Some("Story".to_string()),
|
||||
text_source_kind: Some("text_asset".to_string()),
|
||||
source_url: Some("https://example.invalid/story".to_string()),
|
||||
};
|
||||
TranslationMemoryDraft {
|
||||
source_text: source.to_string(),
|
||||
source_context: translation_memory_context(
|
||||
"Bundles/story.bundle",
|
||||
None,
|
||||
Some("CAB-story"),
|
||||
Some(1),
|
||||
Some(49),
|
||||
Some("m_Text"),
|
||||
Some("plain"),
|
||||
Some("Story"),
|
||||
Some("text_asset"),
|
||||
&TranslationMemoryContext::new(),
|
||||
),
|
||||
translated_text: translated.to_string(),
|
||||
translation_source_kind: TranslationMemorySourceKind::Provider,
|
||||
official_release_id: release.to_string(),
|
||||
source_trace,
|
||||
provider: Some("mock".to_string()),
|
||||
provider_run_id: Some(format!("mock:{release}")),
|
||||
observed_unix_seconds: 1,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn initializes_schema_and_reuses_trusted_entry_across_releases() {
|
||||
let temp = tempfile::TempDir::new().unwrap();
|
||||
let repository = SqliteTranslationMemoryRepository::new(
|
||||
temp.path().join(TRANSLATION_MEMORY_REPOSITORY_FILE),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let entry = repository
|
||||
.upsert_candidate(draft("release-1", "Hello", "你好"))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(repository.summary().await.unwrap().candidate_count, 1);
|
||||
let trusted = repository
|
||||
.confirm(&entry.record_id, "reviewer", Some("accepted".to_string()))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(trusted.trust_status, TranslationMemoryTrustStatus::Trusted);
|
||||
|
||||
let query = draft("release-2", "Hello", "ignored");
|
||||
let matches = repository
|
||||
.find_matches("Hello", &query.source_context, 10)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(matches.len(), 1);
|
||||
assert_eq!(
|
||||
matches[0].match_kind,
|
||||
TranslationMemoryMatchKind::StrongExact
|
||||
);
|
||||
assert!(matches[0].can_auto_reuse);
|
||||
assert_eq!(matches[0].entry.translated_text, "你好");
|
||||
assert_eq!(matches[0].entry.official_release_id, "release-1");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn rejects_future_schema_version() {
|
||||
let temp = tempfile::TempDir::new().unwrap();
|
||||
let path = temp.path().join("tm.sqlite");
|
||||
let repository = SqliteTranslationMemoryRepository::new(&path).await.unwrap();
|
||||
sqlx::query("UPDATE schema_migrations SET version = ?2 WHERE component = ?1")
|
||||
.bind(TRANSLATION_MEMORY_SCHEMA_COMPONENT)
|
||||
.bind(i64::from(TRANSLATION_MEMORY_SCHEMA_VERSION) + 1)
|
||||
.execute(&repository.pool)
|
||||
.await
|
||||
.unwrap();
|
||||
repository.pool.close().await;
|
||||
|
||||
let error = SqliteTranslationMemoryRepository::new(&path)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(error
|
||||
.to_string()
|
||||
.contains("不支持的 Translation Memory schema"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn different_context_is_candidate_only_and_provider_repeat_is_idempotent() {
|
||||
let temp = tempfile::TempDir::new().unwrap();
|
||||
let repository = SqliteTranslationMemoryRepository::new(temp.path().join("tm.sqlite"))
|
||||
.await
|
||||
.unwrap();
|
||||
let first = draft("release-1", "Hello", "你好");
|
||||
repository.upsert_candidate(first.clone()).await.unwrap();
|
||||
repository.upsert_candidate(first).await.unwrap();
|
||||
assert_eq!(repository.summary().await.unwrap().record_count, 1);
|
||||
|
||||
let mut different = draft("release-2", "Hello", "你好");
|
||||
different
|
||||
.source_context
|
||||
.insert("field_path".to_string(), "m_Other".to_string());
|
||||
let matches = repository
|
||||
.find_matches("Hello", &different.source_context, 10)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
matches[0].match_kind,
|
||||
TranslationMemoryMatchKind::CandidateExact
|
||||
);
|
||||
assert!(!matches[0].can_auto_reuse);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn rejects_empty_translation_candidates() {
|
||||
let temp = tempfile::TempDir::new().unwrap();
|
||||
let repository = SqliteTranslationMemoryRepository::new(temp.path().join("tm.sqlite"))
|
||||
.await
|
||||
.unwrap();
|
||||
let error = repository
|
||||
.upsert_candidate(draft("release-1", "Hello", " \n"))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(error.to_string().contains("translated_text"));
|
||||
assert_eq!(repository.summary().await.unwrap().record_count, 0);
|
||||
}
|
||||
}
|
||||
@@ -161,6 +161,12 @@ pub struct TranslationTaskUnitResult {
|
||||
pub source_text: String,
|
||||
/// Provider-produced or human-supplied translation.
|
||||
pub translated_text: String,
|
||||
/// Result source kind.
|
||||
#[serde(default)]
|
||||
pub source_kind: TranslationTaskResultSourceKind,
|
||||
/// Trusted Translation Memory record used for this result, when applicable.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub translation_memory_record_id: Option<String>,
|
||||
/// Provider identifier.
|
||||
pub provider: String,
|
||||
/// Provider run that produced this result.
|
||||
@@ -169,6 +175,30 @@ pub struct TranslationTaskUnitResult {
|
||||
pub translated_unix_seconds: u64,
|
||||
}
|
||||
|
||||
/// Source of one persisted TextUnit translation result.
|
||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum TranslationTaskResultSourceKind {
|
||||
/// Result returned by the configured provider.
|
||||
#[default]
|
||||
Provider,
|
||||
/// Result submitted through the manual task update interface.
|
||||
Manual,
|
||||
/// Result reused from a trusted Translation Memory entry.
|
||||
TranslationMemory,
|
||||
}
|
||||
|
||||
impl TranslationTaskResultSourceKind {
|
||||
/// Returns the stable JSON label.
|
||||
pub const fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Provider => "provider",
|
||||
Self::Manual => "manual",
|
||||
Self::TranslationMemory => "translation_memory",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One provider execution associated with one or more translation units.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct ProviderRun {
|
||||
@@ -247,12 +277,18 @@ pub fn build_translation_handoff(
|
||||
Vec::new(),
|
||||
)
|
||||
});
|
||||
let unit_status = match state.0 {
|
||||
TranslationTaskStatus::Queued => TranslationUnitStatus::Queued,
|
||||
TranslationTaskStatus::Running => TranslationUnitStatus::Translating,
|
||||
TranslationTaskStatus::Failed => TranslationUnitStatus::Failed,
|
||||
TranslationTaskStatus::Completed => TranslationUnitStatus::Translated,
|
||||
TranslationTaskStatus::Skipped => TranslationUnitStatus::Skipped,
|
||||
let unit_status = if state.5.is_empty() {
|
||||
match state.0 {
|
||||
TranslationTaskStatus::Queued => TranslationUnitStatus::Queued,
|
||||
TranslationTaskStatus::Running => TranslationUnitStatus::Translating,
|
||||
TranslationTaskStatus::Failed => TranslationUnitStatus::Failed,
|
||||
TranslationTaskStatus::Completed => TranslationUnitStatus::Translated,
|
||||
TranslationTaskStatus::Skipped => TranslationUnitStatus::Skipped,
|
||||
}
|
||||
} else {
|
||||
// A task can retain successful TM hits while the remaining provider
|
||||
// units are failed or waiting for retry.
|
||||
TranslationUnitStatus::Translated
|
||||
};
|
||||
let unit = TranslationUnit {
|
||||
unit_id: task.task_id.clone(),
|
||||
@@ -1168,11 +1204,29 @@ impl SqliteTranslationTaskRepository {
|
||||
pub async fn fail_claim(
|
||||
&self,
|
||||
failure: TranslationTaskFailure,
|
||||
) -> Result<PersistedTranslationTask> {
|
||||
self.fail_claim_with_results(failure, &[]).await
|
||||
}
|
||||
|
||||
/// Records a provider failure while retaining any already-resolved TextUnit
|
||||
/// results, such as trusted Translation Memory hits.
|
||||
pub async fn fail_claim_with_results(
|
||||
&self,
|
||||
failure: TranslationTaskFailure,
|
||||
translation_results: &[TranslationTaskUnitResult],
|
||||
) -> Result<PersistedTranslationTask> {
|
||||
let now = unix_seconds_now_i64();
|
||||
let next_attempt = failure
|
||||
.next_attempt_unix_seconds
|
||||
.map(|value| i64::try_from(value).unwrap_or(i64::MAX));
|
||||
let translation_results_json = if translation_results.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(
|
||||
serde_json::to_string(translation_results)
|
||||
.map_err(|error| bat_core::Error::Serialization(error.to_string()))?,
|
||||
)
|
||||
};
|
||||
let result = sqlx::query(
|
||||
r#"
|
||||
UPDATE translation_tasks
|
||||
@@ -1183,11 +1237,12 @@ impl SqliteTranslationTaskRepository {
|
||||
lease_expires_unix_seconds = NULL,
|
||||
failure_class = ?4,
|
||||
failure_retryable = ?5,
|
||||
next_attempt_unix_seconds = ?6
|
||||
next_attempt_unix_seconds = ?6,
|
||||
translation_results_json = COALESCE(?7, translation_results_json)
|
||||
WHERE task_id = ?1
|
||||
AND worker_status = 'running'
|
||||
AND lease_owner = ?7
|
||||
AND provider_run_id = ?8
|
||||
AND lease_owner = ?8
|
||||
AND provider_run_id = ?9
|
||||
"#,
|
||||
)
|
||||
.bind(&failure.task_id)
|
||||
@@ -1196,6 +1251,7 @@ impl SqliteTranslationTaskRepository {
|
||||
.bind(&failure.failure_class)
|
||||
.bind(if failure.retryable { 1_i64 } else { 0_i64 })
|
||||
.bind(next_attempt)
|
||||
.bind(translation_results_json)
|
||||
.bind(&failure.worker_id)
|
||||
.bind(&failure.provider_run_id)
|
||||
.execute(&self.pool)
|
||||
@@ -1747,6 +1803,8 @@ mod tests {
|
||||
unit_id: "unit-a".to_string(),
|
||||
source_text: "source".to_string(),
|
||||
translated_text: "manual translation".to_string(),
|
||||
source_kind: TranslationTaskResultSourceKind::Manual,
|
||||
translation_memory_record_id: None,
|
||||
provider: "manual".to_string(),
|
||||
provider_run_id: "manual-run-1".to_string(),
|
||||
translated_unix_seconds: 321,
|
||||
@@ -1864,6 +1922,8 @@ mod tests {
|
||||
unit_id: "unit-a".to_string(),
|
||||
source_text: "source".to_string(),
|
||||
translated_text: "translated".to_string(),
|
||||
source_kind: TranslationTaskResultSourceKind::Provider,
|
||||
translation_memory_record_id: None,
|
||||
provider: "mock".to_string(),
|
||||
provider_run_id: second_run.clone(),
|
||||
translated_unix_seconds: 1,
|
||||
|
||||
@@ -2,15 +2,21 @@
|
||||
//!
|
||||
//! worker 只消费已发布 release 中的 TextUnit 索引和 SQLite 任务状态，不
|
||||
//! 修改官方资源。provider 的输入、输出和错误分类是稳定 contract；状态、
|
||||
//! 租约和译文结果始终写入 `translation-tasks.sqlite`。
|
||||
//! 租约和任务结果写入 release 级 `translation-tasks.sqlite`，跨 release 的
|
||||
//! Translation Memory 写入项目级独立 SQLite 数据库。
|
||||
|
||||
use crate::official_parse::{read_textunit_index_at, OfficialTextUnitIndexUnit};
|
||||
use crate::official_textunit_queue::read_textunit_task_queue_at;
|
||||
use crate::translation_memory::{translation_memory_context, SqliteTranslationMemoryRepository};
|
||||
use crate::translation_tasks::{
|
||||
PersistedTranslationTask, SqliteTranslationTaskRepository, TranslationTaskFailure,
|
||||
TranslationTaskUnitResult,
|
||||
TranslationTaskResultSourceKind, TranslationTaskUnitResult,
|
||||
};
|
||||
use async_trait::async_trait;
|
||||
use bat_core::domain::{
|
||||
TranslationMemoryDraft, TranslationMemorySourceKind, TranslationMemorySourceTrace,
|
||||
};
|
||||
use bat_core::repositories::TranslationMemoryRepository;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
use std::env;
|
||||
@@ -89,6 +95,9 @@ pub struct TranslationWorkerConfig {
|
||||
pub max_tasks: Option<usize>,
|
||||
/// worker 实例前缀，用于 lease 诊断。
|
||||
pub worker_id: String,
|
||||
/// Translation Memory SQLite path. `None` uses the output-root default.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub translation_memory_path: Option<PathBuf>,
|
||||
}
|
||||
|
||||
impl Default for TranslationWorkerConfig {
|
||||
@@ -102,6 +111,7 @@ impl Default for TranslationWorkerConfig {
|
||||
retry_backoff: DEFAULT_TRANSLATION_RETRY_BACKOFF,
|
||||
max_tasks: None,
|
||||
worker_id: format!("bat-worker-{}", std::process::id()),
|
||||
translation_memory_path: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -576,6 +586,16 @@ pub struct TranslationWorkerReport {
|
||||
pub remaining_count: usize,
|
||||
/// 失败诊断。
|
||||
pub failures: Vec<TranslationWorkerFailure>,
|
||||
/// Translation Memory database path used by this run.
|
||||
pub translation_memory_path: PathBuf,
|
||||
/// Whether the Translation Memory repository was available.
|
||||
pub translation_memory_available: bool,
|
||||
/// Number of TextUnits reused from trusted Translation Memory.
|
||||
pub translation_memory_hit_count: usize,
|
||||
/// Number of TextUnits sent to the provider after Translation Memory lookup.
|
||||
pub provider_unit_count: usize,
|
||||
/// Translation Memory diagnostics that did not invalidate provider work.
|
||||
pub translation_memory_failures: Vec<String>,
|
||||
}
|
||||
|
||||
/// worker 失败诊断。
|
||||
@@ -599,7 +619,10 @@ struct WorkerStats {
|
||||
completed_count: AtomicUsize,
|
||||
failed_count: AtomicUsize,
|
||||
retry_scheduled_count: AtomicUsize,
|
||||
translation_memory_hit_count: AtomicUsize,
|
||||
provider_unit_count: AtomicUsize,
|
||||
failures: Mutex<Vec<TranslationWorkerFailure>>,
|
||||
translation_memory_failures: Mutex<Vec<String>>,
|
||||
}
|
||||
|
||||
struct WorkerTaskContext<'a> {
|
||||
@@ -611,12 +634,25 @@ struct WorkerTaskContext<'a> {
|
||||
max_attempts: u32,
|
||||
retry_backoff: Duration,
|
||||
stats: &'a WorkerStats,
|
||||
translation_memory: Option<&'a dyn TranslationMemoryRepository>,
|
||||
}
|
||||
|
||||
/// 运行一个 provider worker 轮次。
|
||||
pub async fn run_translation_worker_at(
|
||||
resource_root: &Path,
|
||||
config: &TranslationWorkerConfig,
|
||||
) -> anyhow::Result<TranslationWorkerReport> {
|
||||
run_translation_worker_at_with_cancellation(resource_root, config, Arc::new(|| false)).await
|
||||
}
|
||||
|
||||
/// 运行一个可协作取消的 provider worker 轮次。
|
||||
///
|
||||
/// 取消只在 claim 循环边界检查；已经开始的 provider 请求会先完成，
|
||||
/// 避免丢失 lease 结果或遗留未清理的外部子进程。
|
||||
pub async fn run_translation_worker_at_with_cancellation(
|
||||
resource_root: &Path,
|
||||
config: &TranslationWorkerConfig,
|
||||
should_cancel: Arc<dyn Fn() -> bool + Send + Sync>,
|
||||
) -> anyhow::Result<TranslationWorkerReport> {
|
||||
config.validate()?;
|
||||
let provider: Arc<dyn TranslationProvider> = match config.provider {
|
||||
@@ -625,7 +661,13 @@ pub async fn run_translation_worker_at(
|
||||
)?),
|
||||
TranslationProviderKind::Crowdin => Arc::new(CrowdinProvider::from_env()?),
|
||||
};
|
||||
run_translation_worker_with_provider(resource_root, config, provider).await
|
||||
run_translation_worker_with_provider_and_cancellation(
|
||||
resource_root,
|
||||
config,
|
||||
provider,
|
||||
should_cancel,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// 使用指定 provider 运行 worker，供测试和插件宿主使用。
|
||||
@@ -633,6 +675,21 @@ pub async fn run_translation_worker_with_provider(
|
||||
resource_root: &Path,
|
||||
config: &TranslationWorkerConfig,
|
||||
provider: Arc<dyn TranslationProvider>,
|
||||
) -> anyhow::Result<TranslationWorkerReport> {
|
||||
run_translation_worker_with_provider_and_cancellation(
|
||||
resource_root,
|
||||
config,
|
||||
provider,
|
||||
Arc::new(|| false),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn run_translation_worker_with_provider_and_cancellation(
|
||||
resource_root: &Path,
|
||||
config: &TranslationWorkerConfig,
|
||||
provider: Arc<dyn TranslationProvider>,
|
||||
should_cancel: Arc<dyn Fn() -> bool + Send + Sync>,
|
||||
) -> anyhow::Result<TranslationWorkerReport> {
|
||||
config.validate()?;
|
||||
let queue = read_textunit_task_queue_at(resource_root)
|
||||
@@ -643,6 +700,21 @@ pub async fn run_translation_worker_with_provider(
|
||||
.map_err(anyhow::Error::msg)?
|
||||
.ok_or_else(|| anyhow::anyhow!("缺少官方 TextUnit 明细索引"))?,
|
||||
);
|
||||
let translation_memory_path = config
|
||||
.translation_memory_path
|
||||
.clone()
|
||||
.unwrap_or_else(|| SqliteTranslationMemoryRepository::repository_path(resource_root));
|
||||
let (translation_memory, translation_memory_startup_failure) =
|
||||
match SqliteTranslationMemoryRepository::new(&translation_memory_path).await {
|
||||
Ok(repository) => (Some(Arc::new(repository)), None),
|
||||
Err(error) => (
|
||||
None,
|
||||
Some(format!(
|
||||
"打开 Translation Memory 数据库失败 {}：{error}",
|
||||
translation_memory_path.display()
|
||||
)),
|
||||
),
|
||||
};
|
||||
let repository = Arc::new(
|
||||
SqliteTranslationTaskRepository::new(SqliteTranslationTaskRepository::repository_path(
|
||||
resource_root,
|
||||
@@ -659,6 +731,13 @@ pub async fn run_translation_worker_with_provider(
|
||||
.await
|
||||
.map_err(|error| anyhow::anyhow!("回收翻译 worker lease 失败：{error}"))?;
|
||||
let stats = Arc::new(WorkerStats::default());
|
||||
if let Some(failure) = translation_memory_startup_failure {
|
||||
stats
|
||||
.translation_memory_failures
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("写入 Translation Memory 诊断时 mutex poisoned"))?
|
||||
.push(failure);
|
||||
}
|
||||
let claimed_limit = Arc::new(AtomicUsize::new(0));
|
||||
let mut handles = Vec::with_capacity(config.concurrency);
|
||||
|
||||
@@ -674,8 +753,13 @@ pub async fn run_translation_worker_with_provider(
|
||||
let max_attempts = config.max_attempts;
|
||||
let lease_seconds = config.lease_seconds;
|
||||
let retry_backoff = config.retry_backoff;
|
||||
let translation_memory = translation_memory.clone();
|
||||
let should_cancel = Arc::clone(&should_cancel);
|
||||
handles.push(tokio::spawn(async move {
|
||||
loop {
|
||||
if should_cancel() {
|
||||
return Err(anyhow::anyhow!("翻译 worker 已取消"));
|
||||
}
|
||||
if let Some(max_tasks) = max_tasks {
|
||||
let reservation = claimed_limit.fetch_add(1, Ordering::AcqRel);
|
||||
if reservation >= max_tasks {
|
||||
@@ -704,6 +788,9 @@ pub async fn run_translation_worker_with_provider(
|
||||
max_attempts,
|
||||
retry_backoff,
|
||||
stats: &stats,
|
||||
translation_memory: translation_memory
|
||||
.as_deref()
|
||||
.map(|repository| repository as &dyn TranslationMemoryRepository),
|
||||
},
|
||||
&task,
|
||||
)
|
||||
@@ -712,10 +799,24 @@ pub async fn run_translation_worker_with_provider(
|
||||
Ok::<(), anyhow::Error>(())
|
||||
}));
|
||||
}
|
||||
let mut first_worker_error = None;
|
||||
for handle in handles {
|
||||
handle
|
||||
.await
|
||||
.map_err(|error| anyhow::anyhow!("等待翻译 worker 失败：{error}"))??;
|
||||
match handle.await {
|
||||
Ok(Ok(())) => {}
|
||||
Ok(Err(error)) => {
|
||||
if first_worker_error.is_none() {
|
||||
first_worker_error = Some(error);
|
||||
}
|
||||
}
|
||||
Err(error) => {
|
||||
if first_worker_error.is_none() {
|
||||
first_worker_error = Some(anyhow::anyhow!("等待翻译 worker 失败：{error}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(error) = first_worker_error {
|
||||
return Err(error);
|
||||
}
|
||||
|
||||
let remaining_count = repository
|
||||
@@ -736,6 +837,11 @@ pub async fn run_translation_worker_with_provider(
|
||||
.map_err(|_| anyhow::anyhow!("读取翻译 worker 失败列表时 mutex poisoned"))?
|
||||
.clone();
|
||||
let failed_count = stats.failed_count.load(Ordering::Relaxed);
|
||||
let translation_memory_failures = stats
|
||||
.translation_memory_failures
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("读取 Translation Memory 诊断时 mutex poisoned"))?
|
||||
.clone();
|
||||
Ok(TranslationWorkerReport {
|
||||
command: "translation-worker",
|
||||
status: if failed_count == 0 {
|
||||
@@ -752,6 +858,11 @@ pub async fn run_translation_worker_with_provider(
|
||||
retry_scheduled_count: stats.retry_scheduled_count.load(Ordering::Relaxed),
|
||||
remaining_count,
|
||||
failures,
|
||||
translation_memory_path,
|
||||
translation_memory_available: translation_memory.is_some(),
|
||||
translation_memory_hit_count: stats.translation_memory_hit_count.load(Ordering::Relaxed),
|
||||
provider_unit_count: stats.provider_unit_count.load(Ordering::Relaxed),
|
||||
translation_memory_failures,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -759,66 +870,176 @@ async fn process_claimed_task(
|
||||
context: &WorkerTaskContext<'_>,
|
||||
task: &PersistedTranslationTask,
|
||||
) -> anyhow::Result<()> {
|
||||
let request = match provider_request(task, context.index) {
|
||||
Ok(request) => request,
|
||||
Err(error) => {
|
||||
record_provider_failure(
|
||||
context,
|
||||
task,
|
||||
TranslationProviderError::new(
|
||||
TranslationProviderFailureClass::InvalidRequest,
|
||||
error.to_string(),
|
||||
),
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
match context.provider.translate(request.clone()).await {
|
||||
Ok(response) => {
|
||||
let results =
|
||||
match validate_provider_response(&request, response, context.provider_name) {
|
||||
Ok(results) => results,
|
||||
Err(error) => {
|
||||
record_provider_failure(
|
||||
context,
|
||||
task,
|
||||
TranslationProviderError::new(
|
||||
TranslationProviderFailureClass::InvalidRequest,
|
||||
error.to_string(),
|
||||
),
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
let task_units = task_index_units(task, context.index)?;
|
||||
let mut results = BTreeMap::new();
|
||||
let mut provider_units = Vec::new();
|
||||
for unit in &task_units {
|
||||
if let Some(translation_memory) = context.translation_memory {
|
||||
let source_context = translation_memory_context(
|
||||
&unit.destination,
|
||||
unit.archive_entry.as_deref(),
|
||||
unit.serialized_file.as_deref(),
|
||||
unit.path_id,
|
||||
unit.class_id,
|
||||
unit.field_path.as_deref(),
|
||||
unit.format.as_deref(),
|
||||
unit.asset_name.as_deref(),
|
||||
unit.text_source_kind.as_deref(),
|
||||
&unit.context,
|
||||
);
|
||||
match translation_memory
|
||||
.find_matches(&unit.source_text, &source_context, 1)
|
||||
.await
|
||||
{
|
||||
Ok(matches) => {
|
||||
if let Some(found) = matches.into_iter().find(|item| item.can_auto_reuse) {
|
||||
context
|
||||
.stats
|
||||
.translation_memory_hit_count
|
||||
.fetch_add(1, Ordering::Relaxed);
|
||||
results.insert(
|
||||
unit.id.clone(),
|
||||
translation_memory_result(task, unit, &found.entry),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
context
|
||||
.repository
|
||||
.complete_claim(
|
||||
&task.task.task_id,
|
||||
context.worker_id,
|
||||
&task.provider_run_id.clone().unwrap_or_default(),
|
||||
context.provider_name,
|
||||
}
|
||||
Err(error) => {
|
||||
record_translation_memory_failure(
|
||||
context,
|
||||
format!(
|
||||
"任务 {} TextUnit {} 查询失败：{}",
|
||||
task.task.task_id, unit.id, error
|
||||
),
|
||||
)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
provider_units.push(*unit);
|
||||
}
|
||||
|
||||
if !provider_units.is_empty() {
|
||||
context
|
||||
.stats
|
||||
.provider_unit_count
|
||||
.fetch_add(provider_units.len(), Ordering::Relaxed);
|
||||
let request = match provider_request(task, &provider_units) {
|
||||
Ok(request) => request,
|
||||
Err(error) => {
|
||||
record_provider_failure(
|
||||
context,
|
||||
task,
|
||||
TranslationProviderError::new(
|
||||
TranslationProviderFailureClass::InvalidRequest,
|
||||
error.to_string(),
|
||||
),
|
||||
&results,
|
||||
)
|
||||
.await
|
||||
.map_err(|error| anyhow::anyhow!("写入翻译任务完成结果失败：{error}"))?;
|
||||
context
|
||||
.stats
|
||||
.completed_count
|
||||
.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
Err(error) => {
|
||||
record_provider_failure(context, task, error).await?;
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
match context.provider.translate(request.clone()).await {
|
||||
Ok(response) => {
|
||||
let provider_results =
|
||||
match validate_provider_response(&request, response, context.provider_name) {
|
||||
Ok(results) => results,
|
||||
Err(error) => {
|
||||
record_provider_failure(
|
||||
context,
|
||||
task,
|
||||
TranslationProviderError::new(
|
||||
TranslationProviderFailureClass::InvalidRequest,
|
||||
error.to_string(),
|
||||
),
|
||||
&results,
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
for result in &provider_results {
|
||||
results.insert(result.unit_id.clone(), result.clone());
|
||||
if let Some(unit) = provider_units.iter().find(|unit| unit.id == result.unit_id)
|
||||
{
|
||||
let source_context = translation_memory_context(
|
||||
&unit.destination,
|
||||
unit.archive_entry.as_deref(),
|
||||
unit.serialized_file.as_deref(),
|
||||
unit.path_id,
|
||||
unit.class_id,
|
||||
unit.field_path.as_deref(),
|
||||
unit.format.as_deref(),
|
||||
unit.asset_name.as_deref(),
|
||||
unit.text_source_kind.as_deref(),
|
||||
&unit.context,
|
||||
);
|
||||
let draft = TranslationMemoryDraft {
|
||||
source_text: result.source_text.clone(),
|
||||
source_context,
|
||||
translated_text: result.translated_text.clone(),
|
||||
translation_source_kind: TranslationMemorySourceKind::Provider,
|
||||
official_release_id: task.task.official_release_id.clone(),
|
||||
source_trace: translation_memory_trace(task, unit),
|
||||
provider: Some(context.provider_name.to_string()),
|
||||
provider_run_id: Some(request.provider_run_id.clone()),
|
||||
observed_unix_seconds: unix_seconds_now(),
|
||||
};
|
||||
if let Some(translation_memory) = context.translation_memory {
|
||||
if let Err(error) = translation_memory.upsert_candidate(draft).await {
|
||||
record_translation_memory_failure(
|
||||
context,
|
||||
format!(
|
||||
"任务 {} TextUnit {} 写入候选失败：{}",
|
||||
task.task.task_id, unit.id, error
|
||||
),
|
||||
)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(error) => {
|
||||
record_provider_failure(context, task, error, &results).await?;
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let ordered_results = task_units
|
||||
.iter()
|
||||
.filter_map(|unit| results.get(&unit.id).cloned())
|
||||
.collect::<Vec<_>>();
|
||||
if ordered_results.len() != task_units.len() {
|
||||
return Err(anyhow::anyhow!(
|
||||
"任务 {} 的译文结果不完整：expected={} actual={}",
|
||||
task.task.task_id,
|
||||
task_units.len(),
|
||||
ordered_results.len()
|
||||
));
|
||||
}
|
||||
context
|
||||
.repository
|
||||
.complete_claim(
|
||||
&task.task.task_id,
|
||||
context.worker_id,
|
||||
&task.provider_run_id.clone().unwrap_or_default(),
|
||||
context.provider_name,
|
||||
&ordered_results,
|
||||
)
|
||||
.await
|
||||
.map_err(|error| anyhow::anyhow!("写入翻译任务完成结果失败：{error}"))?;
|
||||
context
|
||||
.stats
|
||||
.completed_count
|
||||
.fetch_add(1, Ordering::Relaxed);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn provider_request(
|
||||
fn task_index_units<'a>(
|
||||
task: &PersistedTranslationTask,
|
||||
index: &crate::official_parse::OfficialTextUnitIndex,
|
||||
) -> anyhow::Result<TranslationProviderRequest> {
|
||||
index: &'a crate::official_parse::OfficialTextUnitIndex,
|
||||
) -> anyhow::Result<Vec<&'a OfficialTextUnitIndexUnit>> {
|
||||
let parse_entry_key = task
|
||||
.task
|
||||
.parse_entry_key
|
||||
@@ -832,7 +1053,6 @@ fn provider_request(
|
||||
&& unit.destination == task.task.destination
|
||||
&& unit.archive_entry == task.task.archive_entry
|
||||
})
|
||||
.map(|unit| provider_unit(task, unit))
|
||||
.collect::<Vec<_>>();
|
||||
if units.is_empty() {
|
||||
return Err(anyhow::anyhow!(
|
||||
@@ -840,6 +1060,13 @@ fn provider_request(
|
||||
task.task.task_id
|
||||
));
|
||||
}
|
||||
Ok(units)
|
||||
}
|
||||
|
||||
fn provider_request(
|
||||
task: &PersistedTranslationTask,
|
||||
index_units: &[&OfficialTextUnitIndexUnit],
|
||||
) -> anyhow::Result<TranslationProviderRequest> {
|
||||
let provider_run_id = task
|
||||
.provider_run_id
|
||||
.clone()
|
||||
@@ -850,7 +1077,10 @@ fn provider_request(
|
||||
task_id: task.task.task_id.clone(),
|
||||
destination: task.task.destination.clone(),
|
||||
archive_entry: task.task.archive_entry.clone(),
|
||||
units,
|
||||
units: index_units
|
||||
.iter()
|
||||
.map(|unit| provider_unit(task, unit))
|
||||
.collect(),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -914,10 +1144,18 @@ fn validate_provider_response(
|
||||
result.unit_id
|
||||
));
|
||||
}
|
||||
if result.translated_text.trim().is_empty() {
|
||||
return Err(anyhow::anyhow!(
|
||||
"provider 返回空 translated_text：{}",
|
||||
result.unit_id
|
||||
));
|
||||
}
|
||||
results.push(TranslationTaskUnitResult {
|
||||
unit_id: result.unit_id,
|
||||
source_text: result.source_text,
|
||||
translated_text: result.translated_text,
|
||||
source_kind: TranslationTaskResultSourceKind::Provider,
|
||||
translation_memory_record_id: None,
|
||||
provider: provider_name.to_string(),
|
||||
provider_run_id: request.provider_run_id.clone(),
|
||||
translated_unix_seconds: unix_seconds_now(),
|
||||
@@ -933,25 +1171,81 @@ fn validate_provider_response(
|
||||
Ok(results)
|
||||
}
|
||||
|
||||
fn translation_memory_result(
|
||||
task: &PersistedTranslationTask,
|
||||
unit: &OfficialTextUnitIndexUnit,
|
||||
entry: &bat_core::domain::TranslationMemoryEntry,
|
||||
) -> TranslationTaskUnitResult {
|
||||
TranslationTaskUnitResult {
|
||||
unit_id: unit.id.clone(),
|
||||
source_text: unit.source_text.clone(),
|
||||
translated_text: entry.translated_text.clone(),
|
||||
source_kind: TranslationTaskResultSourceKind::TranslationMemory,
|
||||
translation_memory_record_id: Some(entry.record_id.clone()),
|
||||
provider: "translation_memory".to_string(),
|
||||
provider_run_id: task.provider_run_id.clone().unwrap_or_default(),
|
||||
translated_unix_seconds: unix_seconds_now(),
|
||||
}
|
||||
}
|
||||
|
||||
fn translation_memory_trace(
|
||||
task: &PersistedTranslationTask,
|
||||
unit: &OfficialTextUnitIndexUnit,
|
||||
) -> TranslationMemorySourceTrace {
|
||||
TranslationMemorySourceTrace {
|
||||
official_release_id: task.task.official_release_id.clone(),
|
||||
unit_id: Some(unit.id.clone()),
|
||||
task_id: Some(task.task.task_id.clone()),
|
||||
destination: Some(unit.destination.clone()),
|
||||
archive_entry: unit.archive_entry.clone(),
|
||||
serialized_file: unit.serialized_file.clone(),
|
||||
path_id: unit.path_id,
|
||||
class_id: unit.class_id,
|
||||
field_path: unit.field_path.clone(),
|
||||
format: unit.format.clone(),
|
||||
asset_name: unit.asset_name.clone(),
|
||||
text_source_kind: unit.text_source_kind.clone(),
|
||||
source_url: Some(unit.source_url.clone()),
|
||||
}
|
||||
}
|
||||
|
||||
fn record_translation_memory_failure(
|
||||
context: &WorkerTaskContext<'_>,
|
||||
message: String,
|
||||
) -> anyhow::Result<()> {
|
||||
context
|
||||
.stats
|
||||
.translation_memory_failures
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("写入 Translation Memory 诊断时 mutex poisoned"))?
|
||||
.push(message);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn record_provider_failure(
|
||||
context: &WorkerTaskContext<'_>,
|
||||
task: &PersistedTranslationTask,
|
||||
error: TranslationProviderError,
|
||||
partial_results: &BTreeMap<String, TranslationTaskUnitResult>,
|
||||
) -> anyhow::Result<()> {
|
||||
let retryable = error.retryable && task.attempt_count < context.max_attempts;
|
||||
let next_attempt =
|
||||
retryable.then(|| unix_seconds_now().saturating_add(context.retry_backoff.as_secs()));
|
||||
let partial_results = partial_results.values().cloned().collect::<Vec<_>>();
|
||||
context
|
||||
.repository
|
||||
.fail_claim(TranslationTaskFailure {
|
||||
task_id: task.task.task_id.clone(),
|
||||
worker_id: context.worker_id.to_string(),
|
||||
provider_run_id: task.provider_run_id.clone().unwrap_or_default(),
|
||||
failure_class: error.class.as_str().to_string(),
|
||||
failure_reason: error.message.clone(),
|
||||
retryable,
|
||||
next_attempt_unix_seconds: next_attempt,
|
||||
})
|
||||
.fail_claim_with_results(
|
||||
TranslationTaskFailure {
|
||||
task_id: task.task.task_id.clone(),
|
||||
worker_id: context.worker_id.to_string(),
|
||||
provider_run_id: task.provider_run_id.clone().unwrap_or_default(),
|
||||
failure_class: error.class.as_str().to_string(),
|
||||
failure_reason: error.message.clone(),
|
||||
retryable,
|
||||
next_attempt_unix_seconds: next_attempt,
|
||||
},
|
||||
&partial_results,
|
||||
)
|
||||
.await
|
||||
.map_err(|failure| anyhow::anyhow!("写入翻译任务失败状态失败：{failure}"))?;
|
||||
context.stats.failed_count.fetch_add(1, Ordering::Relaxed);
|
||||
@@ -1161,6 +1455,277 @@ mod tests {
|
||||
assert_eq!(task.translation_results[0].translated_text, "translated-0");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn worker_honors_cancellation_before_claiming_tasks() {
|
||||
let (temp, queue) = fixture_root();
|
||||
crate::official_textunit_queue::write_textunit_task_queue_at(temp.path(), &queue).unwrap();
|
||||
crate::official_parse::write_textunit_index_at(temp.path(), &index(temp.path())).unwrap();
|
||||
let config = TranslationWorkerConfig {
|
||||
concurrency: 2,
|
||||
..TranslationWorkerConfig::default()
|
||||
};
|
||||
|
||||
let result =
|
||||
run_translation_worker_at_with_cancellation(temp.path(), &config, Arc::new(|| true))
|
||||
.await;
|
||||
|
||||
assert!(result.unwrap_err().to_string().contains("已取消"));
|
||||
let repository = SqliteTranslationTaskRepository::open(
|
||||
SqliteTranslationTaskRepository::repository_path(temp.path()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let task = repository.find(&queue.tasks[0].task_id).await.unwrap();
|
||||
assert_eq!(
|
||||
task.task_status,
|
||||
crate::translation_tasks::TranslationTaskStatus::Queued
|
||||
);
|
||||
assert_eq!(task.attempt_count, 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn worker_reuses_trusted_tm_for_part_of_a_task_and_calls_provider_for_the_rest() {
|
||||
let (temp, queue) = fixture_root();
|
||||
let textunit_index = index(temp.path());
|
||||
crate::official_textunit_queue::write_textunit_task_queue_at(temp.path(), &queue).unwrap();
|
||||
crate::official_parse::write_textunit_index_at(temp.path(), &textunit_index).unwrap();
|
||||
|
||||
let translation_memory_path = temp.path().join("translation-memory.sqlite");
|
||||
let translation_memory = SqliteTranslationMemoryRepository::new(&translation_memory_path)
|
||||
.await
|
||||
.unwrap();
|
||||
let unit = &textunit_index.units[0];
|
||||
let source_context = translation_memory_context(
|
||||
&unit.destination,
|
||||
unit.archive_entry.as_deref(),
|
||||
unit.serialized_file.as_deref(),
|
||||
unit.path_id,
|
||||
unit.class_id,
|
||||
unit.field_path.as_deref(),
|
||||
unit.format.as_deref(),
|
||||
unit.asset_name.as_deref(),
|
||||
unit.text_source_kind.as_deref(),
|
||||
&unit.context,
|
||||
);
|
||||
let candidate = translation_memory
|
||||
.upsert_candidate(TranslationMemoryDraft {
|
||||
source_text: unit.source_text.clone(),
|
||||
source_context,
|
||||
translated_text: "trusted-from-tm".to_string(),
|
||||
translation_source_kind: TranslationMemorySourceKind::Manual,
|
||||
official_release_id: queue.official_release_id.clone(),
|
||||
source_trace: TranslationMemorySourceTrace {
|
||||
official_release_id: queue.official_release_id.clone(),
|
||||
unit_id: Some(unit.id.clone()),
|
||||
task_id: Some(queue.tasks[0].task_id.clone()),
|
||||
destination: Some(unit.destination.clone()),
|
||||
archive_entry: unit.archive_entry.clone(),
|
||||
serialized_file: unit.serialized_file.clone(),
|
||||
path_id: unit.path_id,
|
||||
class_id: unit.class_id,
|
||||
field_path: unit.field_path.clone(),
|
||||
format: unit.format.clone(),
|
||||
asset_name: unit.asset_name.clone(),
|
||||
text_source_kind: unit.text_source_kind.clone(),
|
||||
source_url: Some(unit.source_url.clone()),
|
||||
},
|
||||
provider: None,
|
||||
provider_run_id: None,
|
||||
observed_unix_seconds: 1,
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
let trusted = translation_memory
|
||||
.confirm(
|
||||
&candidate.record_id,
|
||||
"test-reviewer",
|
||||
Some("accepted".to_string()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
trusted.trust_status,
|
||||
bat_core::domain::TranslationMemoryTrustStatus::Trusted
|
||||
);
|
||||
|
||||
let fixture = temp.path().join("partial-mock.json");
|
||||
std::fs::write(
|
||||
&fixture,
|
||||
serde_json::to_vec(&serde_json::json!({
|
||||
"schema_version": 1,
|
||||
"translations": {
|
||||
"direct:bundle#unit:1": "translated-by-provider"
|
||||
}
|
||||
}))
|
||||
.unwrap(),
|
||||
)
|
||||
.unwrap();
|
||||
let config = TranslationWorkerConfig {
|
||||
fixture_path: Some(fixture),
|
||||
translation_memory_path: Some(translation_memory_path),
|
||||
concurrency: 1,
|
||||
retry_backoff: Duration::ZERO,
|
||||
..TranslationWorkerConfig::default()
|
||||
};
|
||||
let report = run_translation_worker_at(temp.path(), &config)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(report.translation_memory_hit_count, 1);
|
||||
assert_eq!(report.provider_unit_count, 1);
|
||||
assert_eq!(report.completed_count, 1);
|
||||
|
||||
let repository = SqliteTranslationTaskRepository::open(
|
||||
SqliteTranslationTaskRepository::repository_path(temp.path()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let task = repository.find(&queue.tasks[0].task_id).await.unwrap();
|
||||
assert_eq!(task.translation_results.len(), 2);
|
||||
assert_eq!(
|
||||
task.translation_results[0].translated_text,
|
||||
"trusted-from-tm"
|
||||
);
|
||||
assert_eq!(
|
||||
task.translation_results[0].source_kind,
|
||||
TranslationTaskResultSourceKind::TranslationMemory
|
||||
);
|
||||
assert_eq!(
|
||||
task.translation_results[0]
|
||||
.translation_memory_record_id
|
||||
.as_deref(),
|
||||
Some(trusted.record_id.as_str())
|
||||
);
|
||||
assert_eq!(
|
||||
task.translation_results[1].translated_text,
|
||||
"translated-by-provider"
|
||||
);
|
||||
assert_eq!(
|
||||
task.translation_results[1].source_kind,
|
||||
TranslationTaskResultSourceKind::Provider
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn worker_retains_tm_hits_when_provider_fails_for_remaining_units() {
|
||||
let (temp, queue) = fixture_root();
|
||||
let textunit_index = index(temp.path());
|
||||
crate::official_textunit_queue::write_textunit_task_queue_at(temp.path(), &queue).unwrap();
|
||||
crate::official_parse::write_textunit_index_at(temp.path(), &textunit_index).unwrap();
|
||||
|
||||
let translation_memory_path = temp.path().join("translation-memory.sqlite");
|
||||
let translation_memory = SqliteTranslationMemoryRepository::new(&translation_memory_path)
|
||||
.await
|
||||
.unwrap();
|
||||
let unit = &textunit_index.units[0];
|
||||
let source_context = translation_memory_context(
|
||||
&unit.destination,
|
||||
unit.archive_entry.as_deref(),
|
||||
unit.serialized_file.as_deref(),
|
||||
unit.path_id,
|
||||
unit.class_id,
|
||||
unit.field_path.as_deref(),
|
||||
unit.format.as_deref(),
|
||||
unit.asset_name.as_deref(),
|
||||
unit.text_source_kind.as_deref(),
|
||||
&unit.context,
|
||||
);
|
||||
let candidate = translation_memory
|
||||
.upsert_candidate(TranslationMemoryDraft {
|
||||
source_text: unit.source_text.clone(),
|
||||
source_context,
|
||||
translated_text: "trusted-from-tm".to_string(),
|
||||
translation_source_kind: TranslationMemorySourceKind::Manual,
|
||||
official_release_id: queue.official_release_id.clone(),
|
||||
source_trace: TranslationMemorySourceTrace {
|
||||
official_release_id: queue.official_release_id.clone(),
|
||||
unit_id: Some(unit.id.clone()),
|
||||
task_id: Some(queue.tasks[0].task_id.clone()),
|
||||
destination: Some(unit.destination.clone()),
|
||||
archive_entry: unit.archive_entry.clone(),
|
||||
serialized_file: unit.serialized_file.clone(),
|
||||
path_id: unit.path_id,
|
||||
class_id: unit.class_id,
|
||||
field_path: unit.field_path.clone(),
|
||||
format: unit.format.clone(),
|
||||
asset_name: unit.asset_name.clone(),
|
||||
text_source_kind: unit.text_source_kind.clone(),
|
||||
source_url: Some(unit.source_url.clone()),
|
||||
},
|
||||
provider: None,
|
||||
provider_run_id: None,
|
||||
observed_unix_seconds: 1,
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
let trusted = translation_memory
|
||||
.confirm(
|
||||
&candidate.record_id,
|
||||
"test-reviewer",
|
||||
Some("accepted".to_string()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let fixture = temp.path().join("provider-failure.json");
|
||||
std::fs::write(
|
||||
&fixture,
|
||||
serde_json::to_vec(&serde_json::json!({
|
||||
"schema_version": 1,
|
||||
"failures": {
|
||||
"textunit/release-1/bundle": {
|
||||
"class": "rate_limited",
|
||||
"message": "fixture throttled",
|
||||
"retryable": false
|
||||
}
|
||||
}
|
||||
}))
|
||||
.unwrap(),
|
||||
)
|
||||
.unwrap();
|
||||
let config = TranslationWorkerConfig {
|
||||
fixture_path: Some(fixture),
|
||||
translation_memory_path: Some(translation_memory_path),
|
||||
concurrency: 1,
|
||||
max_attempts: 1,
|
||||
retry_backoff: Duration::ZERO,
|
||||
..TranslationWorkerConfig::default()
|
||||
};
|
||||
let report = run_translation_worker_at(temp.path(), &config)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(report.translation_memory_hit_count, 1);
|
||||
assert_eq!(report.provider_unit_count, 1);
|
||||
assert_eq!(report.failed_count, 1);
|
||||
|
||||
let repository = SqliteTranslationTaskRepository::open(
|
||||
SqliteTranslationTaskRepository::repository_path(temp.path()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let task = repository.find(&queue.tasks[0].task_id).await.unwrap();
|
||||
assert_eq!(
|
||||
task.task_status,
|
||||
crate::translation_tasks::TranslationTaskStatus::Failed
|
||||
);
|
||||
assert_eq!(task.translation_results.len(), 1);
|
||||
assert_eq!(
|
||||
task.translation_results[0]
|
||||
.translation_memory_record_id
|
||||
.as_deref(),
|
||||
Some(trusted.record_id.as_str())
|
||||
);
|
||||
|
||||
let handoff = crate::translation_tasks::build_translation_handoff(&queue, &[task]);
|
||||
assert_eq!(
|
||||
handoff.units[0].status,
|
||||
crate::translation_tasks::TranslationUnitStatus::Translated
|
||||
);
|
||||
assert_eq!(
|
||||
handoff.job.status,
|
||||
crate::translation_tasks::TranslationJobStatus::Failed
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mock_worker_retries_retryable_failures_and_keeps_diagnostic() {
|
||||
let (temp, queue) = fixture_root();
|
||||
|
||||
@@ -9,7 +9,7 @@ use crate::path_security::{
|
||||
use crate::{
|
||||
LocalizedPatchInput, LocalizedPatchOperationMetadata, LocalizedStringFieldPatch,
|
||||
LocalizedTextAssetPatch, PersistedTranslationTask, SqliteTranslationTaskRepository,
|
||||
TranslationTaskStatus, TranslationTaskUnitResult,
|
||||
TranslationTaskResultSourceKind, TranslationTaskStatus, TranslationTaskUnitResult,
|
||||
};
|
||||
use bat_assetbundle::{
|
||||
patch_unityfs_field, patch_unityfs_string_field, patch_unityfs_text_asset, FieldPatch,
|
||||
@@ -72,6 +72,12 @@ pub struct TranslationWorkbenchEntry {
|
||||
/// Provider run that produced this translation, when imported from worker output.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub provider_run_id: Option<String>,
|
||||
/// Source of the worker result (`provider`, `manual`, or `translation_memory`).
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub translation_source_kind: Option<String>,
|
||||
/// Trusted Translation Memory record used for this translation, when applicable.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub translation_memory_record_id: Option<String>,
|
||||
/// Worker completion time for provider-produced text.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub translated_unix_seconds: Option<u64>,
|
||||
@@ -616,6 +622,8 @@ fn localized_patch_metadata(entry: &TranslationWorkbenchEntry) -> LocalizedPatch
|
||||
.to_string(),
|
||||
translation_provider: entry.translation_provider.clone(),
|
||||
provider_run_id: entry.provider_run_id.clone(),
|
||||
translation_source_kind: entry.translation_source_kind.clone(),
|
||||
translation_memory_record_id: entry.translation_memory_record_id.clone(),
|
||||
review_status: entry
|
||||
.review_status
|
||||
.clone()
|
||||
@@ -887,18 +895,30 @@ fn workbench_entry_from_worker_result(
|
||||
) -> TranslationWorkbenchEntry {
|
||||
let mut entry = TranslationWorkbenchEntry::from_index(unit);
|
||||
entry.translated_text = Some(result.translated_text.clone());
|
||||
entry.translation_provider = task
|
||||
.provider
|
||||
.clone()
|
||||
.or_else(|| Some(result.provider.clone()))
|
||||
.filter(|provider| !provider.trim().is_empty());
|
||||
entry.translation_provider = match result.source_kind {
|
||||
TranslationTaskResultSourceKind::TranslationMemory => None,
|
||||
TranslationTaskResultSourceKind::Provider | TranslationTaskResultSourceKind::Manual => task
|
||||
.provider
|
||||
.clone()
|
||||
.or_else(|| Some(result.provider.clone()))
|
||||
.filter(|provider| !provider.trim().is_empty()),
|
||||
};
|
||||
entry.provider_run_id = task
|
||||
.provider_run_id
|
||||
.clone()
|
||||
.or_else(|| Some(result.provider_run_id.clone()))
|
||||
.filter(|provider_run_id| !provider_run_id.trim().is_empty());
|
||||
entry.translation_source_kind = Some(result.source_kind.as_str().to_string());
|
||||
entry.translation_memory_record_id = result.translation_memory_record_id.clone();
|
||||
entry.translated_unix_seconds = Some(result.translated_unix_seconds);
|
||||
entry.review_status = Some("provider_completed".to_string());
|
||||
entry.review_status = Some(
|
||||
match result.source_kind {
|
||||
TranslationTaskResultSourceKind::Provider => "provider_completed",
|
||||
TranslationTaskResultSourceKind::Manual => "manual_submitted",
|
||||
TranslationTaskResultSourceKind::TranslationMemory => "translation_memory_reused",
|
||||
}
|
||||
.to_string(),
|
||||
);
|
||||
entry
|
||||
}
|
||||
|
||||
@@ -938,6 +958,8 @@ impl TranslationWorkbenchEntry {
|
||||
translated_text: None,
|
||||
translation_provider: None,
|
||||
provider_run_id: None,
|
||||
translation_source_kind: None,
|
||||
translation_memory_record_id: None,
|
||||
translated_unix_seconds: None,
|
||||
review_status: None,
|
||||
format: unit.format.clone(),
|
||||
@@ -975,6 +997,8 @@ mod tests {
|
||||
translated_text: None,
|
||||
translation_provider: None,
|
||||
provider_run_id: None,
|
||||
translation_source_kind: None,
|
||||
translation_memory_record_id: None,
|
||||
translated_unix_seconds: None,
|
||||
review_status: None,
|
||||
format: Some("plain".to_string()),
|
||||
|
||||
@@ -54,6 +54,8 @@ func (s *Server) handleAdminIndex(w http.ResponseWriter, r *http.Request) {
|
||||
"/admin/parse/errors",
|
||||
"/admin/translation/tasks",
|
||||
"/admin/translation/handoff",
|
||||
"/admin/translation/memory/summary",
|
||||
"/admin/translation/memory/query",
|
||||
"/admin/translation/status",
|
||||
},
|
||||
Controls: []string{
|
||||
@@ -72,6 +74,7 @@ func (s *Server) handleAdminIndex(w http.ResponseWriter, r *http.Request) {
|
||||
"/admin/control/translation-task-update",
|
||||
"/admin/control/translation-worker-run",
|
||||
"/admin/control/translation-proofread",
|
||||
"/admin/control/translation-memory-confirm",
|
||||
"/admin/control/localized-publish",
|
||||
"/admin/control/localized-rollback",
|
||||
},
|
||||
@@ -117,6 +120,10 @@ func (s *Server) handleAdminControl(w http.ResponseWriter, r *http.Request) {
|
||||
s.handleAdminTranslationProofread(w, r)
|
||||
return
|
||||
}
|
||||
if action == "translation-memory-confirm" {
|
||||
s.handleAdminTranslationMemoryConfirm(w, r)
|
||||
return
|
||||
}
|
||||
if action == "localized-publish" {
|
||||
s.handleAdminLocalizedPublish(w, r)
|
||||
return
|
||||
@@ -274,6 +281,34 @@ func (s *Server) handleAdminTranslationProofread(w http.ResponseWriter, r *http.
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) handleAdminTranslationMemoryConfirm(w http.ResponseWriter, r *http.Request) {
|
||||
backend, ok := s.backend.(TranslationMemoryBackend)
|
||||
if !ok || backend == nil {
|
||||
writeErrorJSON(w, http.StatusServiceUnavailable, "translation_memory_backend_unavailable", "Rust bat Translation Memory backend is unavailable")
|
||||
return
|
||||
}
|
||||
var params backendrpc.TranslationMemoryConfirmParams
|
||||
if !decodeAdminTranslationJSON(w, r, ¶ms) {
|
||||
return
|
||||
}
|
||||
if err := validateTranslationMemoryConfirmParams(params); err != nil {
|
||||
writeErrorJSON(w, http.StatusBadRequest, "invalid_translation_memory_params", err.Error())
|
||||
return
|
||||
}
|
||||
result, err := backend.TranslationMemoryConfirm(r.Context(), params)
|
||||
if err != nil {
|
||||
s.writeControlBackendError(w, "translation-memory-confirm", err)
|
||||
return
|
||||
}
|
||||
writeNoStoreJSON(w, http.StatusAccepted, AdminControlResponse{
|
||||
Service: "bat-api",
|
||||
Action: "translation-memory-confirm",
|
||||
RPCMethod: "translation.memory.confirm",
|
||||
Status: "accepted",
|
||||
Result: result,
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) handleAdminLocalizedPublish(w http.ResponseWriter, r *http.Request) {
|
||||
backend, ok := s.backend.(LocalizedBackend)
|
||||
if !ok || backend == nil {
|
||||
@@ -656,6 +691,66 @@ func (s *Server) handleAdminTranslationHandoff(w http.ResponseWriter, r *http.Re
|
||||
writeNoStoreJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
func (s *Server) handleAdminTranslationMemorySummary(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet && r.Method != http.MethodHead {
|
||||
writeErrorJSON(w, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed")
|
||||
return
|
||||
}
|
||||
if !s.requireAdminToken(w, r) {
|
||||
return
|
||||
}
|
||||
backend, ok := s.backend.(TranslationMemoryBackend)
|
||||
if !ok || backend == nil {
|
||||
writeErrorJSON(w, http.StatusServiceUnavailable, "translation_memory_backend_unavailable", "Rust bat Translation Memory backend is unavailable")
|
||||
return
|
||||
}
|
||||
params := backendrpc.TranslationMemorySummaryParams{
|
||||
TranslationMemoryPath: firstTrimmedQuery(r.URL.Query(), "translation_memory_path", "tm_path"),
|
||||
}
|
||||
result, err := backend.TranslationMemorySummary(r.Context(), params)
|
||||
if err != nil {
|
||||
s.writeControlBackendError(w, "translation-memory-summary", err)
|
||||
return
|
||||
}
|
||||
if r.Method == http.MethodHead {
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
writeNoStoreJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
func (s *Server) handleAdminTranslationMemoryQuery(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet && r.Method != http.MethodHead {
|
||||
writeErrorJSON(w, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed")
|
||||
return
|
||||
}
|
||||
if !s.requireAdminToken(w, r) {
|
||||
return
|
||||
}
|
||||
backend, ok := s.backend.(TranslationMemoryBackend)
|
||||
if !ok || backend == nil {
|
||||
writeErrorJSON(w, http.StatusServiceUnavailable, "translation_memory_backend_unavailable", "Rust bat Translation Memory backend is unavailable")
|
||||
return
|
||||
}
|
||||
params, err := translationMemoryQueryParams(r)
|
||||
if err != nil {
|
||||
writeErrorJSON(w, http.StatusBadRequest, "invalid_translation_memory_query", err.Error())
|
||||
return
|
||||
}
|
||||
result, err := backend.TranslationMemoryQuery(r.Context(), params)
|
||||
if err != nil {
|
||||
s.writeControlBackendError(w, "translation-memory-query", err)
|
||||
return
|
||||
}
|
||||
if r.Method == http.MethodHead {
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
writeNoStoreJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
func (s *Server) handleAdminSchedules(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet && r.Method != http.MethodHead {
|
||||
writeErrorJSON(w, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed")
|
||||
@@ -747,6 +842,38 @@ func translationTaskListParams(r *http.Request) (backendrpc.TranslationTaskListP
|
||||
return params, nil
|
||||
}
|
||||
|
||||
func translationMemoryQueryParams(r *http.Request) (backendrpc.TranslationMemoryQueryParams, error) {
|
||||
query := r.URL.Query()
|
||||
sourceText := strings.TrimSpace(query.Get("source_text"))
|
||||
if sourceText == "" {
|
||||
return backendrpc.TranslationMemoryQueryParams{}, errors.New("source_text is required")
|
||||
}
|
||||
params := backendrpc.TranslationMemoryQueryParams{
|
||||
TranslationMemoryPath: firstTrimmedQuery(query, "translation_memory_path", "tm_path"),
|
||||
SourceText: sourceText,
|
||||
}
|
||||
if raw := firstTrimmedQuery(query, "source_context", "context"); raw != "" {
|
||||
var context backendrpc.TranslationMemoryContext
|
||||
if raw != "null" {
|
||||
if err := json.Unmarshal([]byte(raw), &context); err != nil {
|
||||
return backendrpc.TranslationMemoryQueryParams{}, errors.New("source_context must be a JSON object with string values")
|
||||
}
|
||||
if context == nil {
|
||||
return backendrpc.TranslationMemoryQueryParams{}, errors.New("source_context must be a JSON object")
|
||||
}
|
||||
params.SourceContext = context
|
||||
}
|
||||
}
|
||||
if raw := strings.TrimSpace(query.Get("limit")); raw != "" {
|
||||
limit, err := strconv.ParseUint(raw, 10, 64)
|
||||
if err != nil || limit == 0 || limit > 1000 {
|
||||
return backendrpc.TranslationMemoryQueryParams{}, errors.New("limit must be in 1..=1000")
|
||||
}
|
||||
params.Limit = &limit
|
||||
}
|
||||
return params, nil
|
||||
}
|
||||
|
||||
func parseTextUnitQueryParams(r *http.Request) (backendrpc.TextUnitQueryParams, error) {
|
||||
query := r.URL.Query()
|
||||
params := backendrpc.TextUnitQueryParams{
|
||||
@@ -938,6 +1065,13 @@ func validateTranslationWorkerRunParams(params backendrpc.TranslationWorkerRunPa
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateTranslationMemoryConfirmParams(params backendrpc.TranslationMemoryConfirmParams) error {
|
||||
if strings.TrimSpace(params.RecordID) == "" || strings.TrimSpace(params.Reviewer) == "" {
|
||||
return errors.New("Translation Memory confirm requires record_id and reviewer")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateLocalizedPublishParams(params backendrpc.LocalizedPublishParams) error {
|
||||
hasFile := strings.TrimSpace(params.TranslationFile) != ""
|
||||
if hasFile == params.FromWorker {
|
||||
|
||||
+110
-8
@@ -629,13 +629,16 @@ func (f *fakeBackend) TaskCancel(ctx context.Context, taskID string) (*backendrp
|
||||
|
||||
type controlBackend struct {
|
||||
*fakeBackend
|
||||
calls []string
|
||||
parseTextUnitQueries []backendrpc.TextUnitQueryParams
|
||||
parseErrorQueries []backendrpc.TextUnitQueryParams
|
||||
translationTaskUpdates []backendrpc.TranslationTaskUpdateParams
|
||||
translationTaskListParams []backendrpc.TranslationTaskListParams
|
||||
localizedPublishParams []backendrpc.LocalizedPublishParams
|
||||
localizedRollbackParams []backendrpc.LocalizedRollbackParams
|
||||
calls []string
|
||||
parseTextUnitQueries []backendrpc.TextUnitQueryParams
|
||||
parseErrorQueries []backendrpc.TextUnitQueryParams
|
||||
translationTaskUpdates []backendrpc.TranslationTaskUpdateParams
|
||||
translationTaskListParams []backendrpc.TranslationTaskListParams
|
||||
translationMemorySummaryParams []backendrpc.TranslationMemorySummaryParams
|
||||
translationMemoryQueryParams []backendrpc.TranslationMemoryQueryParams
|
||||
translationMemoryConfirmParams []backendrpc.TranslationMemoryConfirmParams
|
||||
localizedPublishParams []backendrpc.LocalizedPublishParams
|
||||
localizedRollbackParams []backendrpc.LocalizedRollbackParams
|
||||
}
|
||||
|
||||
func (b *controlBackend) DaemonReload(ctx context.Context) (*backendrpc.Ack, error) {
|
||||
@@ -777,6 +780,51 @@ func (b *controlBackend) TranslationProofread(ctx context.Context) (json.RawMess
|
||||
return json.RawMessage(`{"translation_workflow_status":"manual_proofreading"}`), nil
|
||||
}
|
||||
|
||||
func (b *controlBackend) TranslationMemorySummary(ctx context.Context, params backendrpc.TranslationMemorySummaryParams) (*backendrpc.TranslationMemorySummaryReport, error) {
|
||||
b.calls = append(b.calls, "translation.memory.summary")
|
||||
b.translationMemorySummaryParams = append(b.translationMemorySummaryParams, params)
|
||||
schemaVersion := uint64(1)
|
||||
return &backendrpc.TranslationMemorySummaryReport{
|
||||
Available: true,
|
||||
Path: params.TranslationMemoryPath,
|
||||
SchemaVersion: &schemaVersion,
|
||||
Summary: &backendrpc.TranslationMemorySummary{
|
||||
SchemaVersion: schemaVersion,
|
||||
RecordCount: 2,
|
||||
TrustedCount: 1,
|
||||
CandidateCount: 1,
|
||||
SupersededCount: 0,
|
||||
RejectedCount: 0,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (b *controlBackend) TranslationMemoryQuery(ctx context.Context, params backendrpc.TranslationMemoryQueryParams) (*backendrpc.TranslationMemoryQueryReport, error) {
|
||||
b.calls = append(b.calls, "translation.memory.query")
|
||||
b.translationMemoryQueryParams = append(b.translationMemoryQueryParams, params)
|
||||
return &backendrpc.TranslationMemoryQueryReport{
|
||||
Available: true,
|
||||
Path: params.TranslationMemoryPath,
|
||||
SourceText: params.SourceText,
|
||||
SourceContext: params.SourceContext,
|
||||
Matches: []backendrpc.TranslationMemoryMatch{},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (b *controlBackend) TranslationMemoryConfirm(ctx context.Context, params backendrpc.TranslationMemoryConfirmParams) (*backendrpc.TranslationMemoryConfirmReport, error) {
|
||||
b.calls = append(b.calls, "translation.memory.confirm")
|
||||
b.translationMemoryConfirmParams = append(b.translationMemoryConfirmParams, params)
|
||||
return &backendrpc.TranslationMemoryConfirmReport{
|
||||
Available: true,
|
||||
Path: params.TranslationMemoryPath,
|
||||
Entry: backendrpc.TranslationMemoryEntry{
|
||||
RecordID: params.RecordID,
|
||||
TranslationSourceKind: "provider",
|
||||
TrustStatus: "trusted",
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (b *controlBackend) LocalizedStatus(ctx context.Context) (json.RawMessage, error) {
|
||||
b.calls = append(b.calls, "localized.status")
|
||||
return json.RawMessage(`{"localized_release_status":"localized","status_code":"localized.published"}`), nil
|
||||
@@ -960,6 +1008,42 @@ func TestAdminTranslationQueryEndpointsProxyAuthenticatedRequests(t *testing.T)
|
||||
t.Fatalf("calls=%v", backend.calls)
|
||||
}
|
||||
|
||||
request = httptest.NewRequest(http.MethodGet, "/admin/translation/memory/summary?translation_memory_path=%2Fvar%2Flib%2Fbat%2Ftranslation-memory.sqlite", nil)
|
||||
request.Header.Set("Authorization", "Bearer translation-token")
|
||||
recorder = httptest.NewRecorder()
|
||||
s.Handler().ServeHTTP(recorder, request)
|
||||
if recorder.Code != http.StatusOK {
|
||||
t.Fatalf("TM summary status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||
}
|
||||
if !strings.Contains(recorder.Body.String(), `"trusted_count":1`) ||
|
||||
len(backend.translationMemorySummaryParams) != 1 ||
|
||||
backend.translationMemorySummaryParams[0].TranslationMemoryPath != "/var/lib/bat/translation-memory.sqlite" {
|
||||
t.Fatalf("TM summary body=%s params=%#v", recorder.Body.String(), backend.translationMemorySummaryParams)
|
||||
}
|
||||
|
||||
request = httptest.NewRequest(http.MethodGet, "/admin/translation/memory/query?source_text=Hello&source_context=%7B%22destination%22%3A%22Bundle%2Fdialogue.bundle%22%7D&limit=25", nil)
|
||||
request.Header.Set("Authorization", "Bearer translation-token")
|
||||
recorder = httptest.NewRecorder()
|
||||
s.Handler().ServeHTTP(recorder, request)
|
||||
if recorder.Code != http.StatusOK {
|
||||
t.Fatalf("TM query status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||
}
|
||||
if len(backend.translationMemoryQueryParams) != 1 ||
|
||||
backend.translationMemoryQueryParams[0].SourceText != "Hello" ||
|
||||
backend.translationMemoryQueryParams[0].SourceContext["destination"] != "Bundle/dialogue.bundle" ||
|
||||
backend.translationMemoryQueryParams[0].Limit == nil ||
|
||||
*backend.translationMemoryQueryParams[0].Limit != 25 {
|
||||
t.Fatalf("TM query body=%s params=%#v", recorder.Body.String(), backend.translationMemoryQueryParams)
|
||||
}
|
||||
|
||||
request = httptest.NewRequest(http.MethodGet, "/admin/translation/memory/query", nil)
|
||||
request.Header.Set("Authorization", "Bearer translation-token")
|
||||
recorder = httptest.NewRecorder()
|
||||
s.Handler().ServeHTTP(recorder, request)
|
||||
if recorder.Code != http.StatusBadRequest {
|
||||
t.Fatalf("missing TM query source status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||
}
|
||||
|
||||
request = httptest.NewRequest(http.MethodGet, "/admin/translation/tasks?limit=0", nil)
|
||||
request.Header.Set("Authorization", "Bearer translation-token")
|
||||
recorder = httptest.NewRecorder()
|
||||
@@ -1502,7 +1586,9 @@ func TestOpenAPIAndAdminReservedEndpoints(t *testing.T) {
|
||||
!strings.Contains(rr.Body.String(), "/admin/tasks") ||
|
||||
!strings.Contains(rr.Body.String(), "/admin/parse/text-units") ||
|
||||
!strings.Contains(rr.Body.String(), "translation_results") ||
|
||||
!strings.Contains(rr.Body.String(), "task-cancel") {
|
||||
!strings.Contains(rr.Body.String(), "task-cancel") ||
|
||||
!strings.Contains(rr.Body.String(), "/admin/translation/memory/query") ||
|
||||
!strings.Contains(rr.Body.String(), "translation-memory-confirm") {
|
||||
t.Fatalf("openapi missing dashboard/task admin routes")
|
||||
}
|
||||
|
||||
@@ -1524,6 +1610,8 @@ func TestOpenAPIAndAdminReservedEndpoints(t *testing.T) {
|
||||
links := strings.Join(admin.Links, "\n")
|
||||
if !strings.Contains(links, "/admin/translation/tasks") ||
|
||||
!strings.Contains(links, "/admin/translation/handoff") ||
|
||||
!strings.Contains(links, "/admin/translation/memory/summary") ||
|
||||
!strings.Contains(links, "/admin/translation/memory/query") ||
|
||||
!strings.Contains(links, "/admin/dashboard/") ||
|
||||
!strings.Contains(links, "/admin/parse/text-units") ||
|
||||
!strings.Contains(links, "/admin/tasks/logs") {
|
||||
@@ -1729,6 +1817,7 @@ func TestAdminControlForwardsAllowlistedActions(t *testing.T) {
|
||||
{name: "translation task update", action: "translation-task-update", body: `{"task_id":"textunit/v-current/Scenario","status":"completed","provider":"manual","provider_run_id":"manual-run-1","translation_results":[{"unit_id":"direct:a#unit:0","source_text":"source","translated_text":"译文"}]}`, rpcMethod: "translation.task.update", call: "translation.task.update"},
|
||||
{name: "translation worker run", action: "translation-worker-run", body: `{"provider":"mock","concurrency":8,"max_tasks":2,"retry_backoff_seconds":0,"worker_id":"dashboard-worker"}`, rpcMethod: "translation.worker.run", call: "translation.worker.run"},
|
||||
{name: "translation proofread", action: "translation-proofread", rpcMethod: "translation.proofread", call: "translation.proofread"},
|
||||
{name: "translation memory confirm", action: "translation-memory-confirm", body: `{"record_id":"tm-record-1","reviewer":"reviewer","reason":"reviewed"}`, rpcMethod: "translation.memory.confirm", call: "translation.memory.confirm"},
|
||||
{name: "localized publish", action: "localized-publish", body: `{"from_worker":true,"localized_release_id":"localized-1","force":true}`, rpcMethod: "localized.publish", call: "localized.publish"},
|
||||
{name: "localized rollback", action: "localized-rollback", body: `{"localized_release_id":"localized-1"}`, rpcMethod: "localized.rollback", call: "localized.rollback"},
|
||||
}
|
||||
@@ -1760,6 +1849,11 @@ func TestAdminControlForwardsAllowlistedActions(t *testing.T) {
|
||||
backend.translationTaskUpdates[0].TranslationResults[0].TranslatedText != "译文" {
|
||||
t.Fatalf("translation task updates=%#v", backend.translationTaskUpdates)
|
||||
}
|
||||
if len(backend.translationMemoryConfirmParams) != 1 ||
|
||||
backend.translationMemoryConfirmParams[0].RecordID != "tm-record-1" ||
|
||||
backend.translationMemoryConfirmParams[0].Reviewer != "reviewer" {
|
||||
t.Fatalf("TM confirm params=%#v", backend.translationMemoryConfirmParams)
|
||||
}
|
||||
|
||||
request := httptest.NewRequest(http.MethodPost, "/admin/control/translation-task-update", strings.NewReader(`{"task_id":""}`))
|
||||
request.Header.Set("Authorization", "Bearer control-token")
|
||||
@@ -1785,6 +1879,14 @@ func TestAdminControlForwardsAllowlistedActions(t *testing.T) {
|
||||
t.Fatalf("invalid translation worker status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||
}
|
||||
|
||||
request = httptest.NewRequest(http.MethodPost, "/admin/control/translation-memory-confirm", strings.NewReader(`{"record_id":""}`))
|
||||
request.Header.Set("Authorization", "Bearer control-token")
|
||||
recorder = httptest.NewRecorder()
|
||||
s.Handler().ServeHTTP(recorder, request)
|
||||
if recorder.Code != http.StatusBadRequest {
|
||||
t.Fatalf("invalid TM confirm status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||
}
|
||||
|
||||
request = httptest.NewRequest(http.MethodPost, "/admin/control/localized-publish", strings.NewReader(`{"from_worker":true,"translation_file":"/tmp/workbench.json"}`))
|
||||
request.Header.Set("Authorization", "Bearer control-token")
|
||||
recorder = httptest.NewRecorder()
|
||||
|
||||
@@ -110,3 +110,91 @@ func TestRustContractFixturesPreserveGoMirror(t *testing.T) {
|
||||
t.Fatalf("legacy game_main_config field unexpectedly present: %s", snapshot.LegacyGameMainConfig)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTranslationMemoryRustContractMirror(t *testing.T) {
|
||||
raw := []byte(`{
|
||||
"available": true,
|
||||
"path": "${TM_PATH}",
|
||||
"source_text": "${SOURCE_TEXT}",
|
||||
"source_context": {
|
||||
"destination": "${DESTINATION}",
|
||||
"field_path": "${FIELD_PATH}"
|
||||
},
|
||||
"matches": [{
|
||||
"entry": {
|
||||
"record_id": "${RECORD_ID}",
|
||||
"source_text": "${SOURCE_TEXT}",
|
||||
"source_hash": "${SOURCE_HASH}",
|
||||
"normalized_source_text": "${NORMALIZED_SOURCE_TEXT}",
|
||||
"source_context": {
|
||||
"destination": "${DESTINATION}",
|
||||
"field_path": "${FIELD_PATH}"
|
||||
},
|
||||
"source_context_hash": "${SOURCE_CONTEXT_HASH}",
|
||||
"translated_text": "${TRANSLATED_TEXT}",
|
||||
"translation_source_kind": "provider",
|
||||
"trust_status": "trusted",
|
||||
"official_release_id": "${OFFICIAL_RELEASE_ID}",
|
||||
"source_trace": {
|
||||
"official_release_id": "${OFFICIAL_RELEASE_ID}",
|
||||
"unit_id": "${UNIT_ID}",
|
||||
"task_id": "${TASK_ID}",
|
||||
"destination": "${DESTINATION}",
|
||||
"archive_entry": "${ARCHIVE_ENTRY}",
|
||||
"serialized_file": "${SERIALIZED_FILE}",
|
||||
"path_id": 42,
|
||||
"class_id": 114,
|
||||
"field_path": "${FIELD_PATH}",
|
||||
"format": "json",
|
||||
"asset_name": "${ASSET_NAME}",
|
||||
"text_source_kind": "text_asset"
|
||||
},
|
||||
"provider": "${PROVIDER}",
|
||||
"provider_run_id": "${PROVIDER_RUN_ID}",
|
||||
"created_unix_seconds": 100,
|
||||
"updated_unix_seconds": 200,
|
||||
"trusted_unix_seconds": 200,
|
||||
"trusted_by": "${REVIEWER}",
|
||||
"trusted_reason": "${TRUST_REASON}"
|
||||
},
|
||||
"match_kind": "strong_exact",
|
||||
"can_auto_reuse": true
|
||||
}]
|
||||
}`)
|
||||
if bytes.Contains(raw, []byte("/tmp/")) {
|
||||
t.Fatal("TM mirror contains a local temporary path")
|
||||
}
|
||||
|
||||
var report backendrpc.TranslationMemoryQueryReport
|
||||
if err := json.Unmarshal(raw, &report); err != nil {
|
||||
t.Fatalf("decode TM query mirror: %v", err)
|
||||
}
|
||||
if !report.Available || report.Path != "${TM_PATH}" ||
|
||||
report.SourceText != "${SOURCE_TEXT}" ||
|
||||
report.SourceContext["field_path"] != "${FIELD_PATH}" ||
|
||||
len(report.Matches) != 1 {
|
||||
t.Fatalf("TM report=%+v", report)
|
||||
}
|
||||
match := report.Matches[0]
|
||||
if match.MatchKind != "strong_exact" || !match.CanAutoReuse ||
|
||||
match.Entry.TrustStatus != "trusted" ||
|
||||
match.Entry.TranslationSourceKind != "provider" ||
|
||||
match.Entry.SourceTrace.PathID == nil || *match.Entry.SourceTrace.PathID != 42 ||
|
||||
match.Entry.SourceTrace.ClassID == nil || *match.Entry.SourceTrace.ClassID != 114 ||
|
||||
match.Entry.TrustedBy == nil || *match.Entry.TrustedBy != "${REVIEWER}" {
|
||||
t.Fatalf("TM match=%+v", match)
|
||||
}
|
||||
|
||||
var missing backendrpc.TranslationMemorySummaryReport
|
||||
if err := json.Unmarshal([]byte(`{
|
||||
"available": false,
|
||||
"path": "${TM_PATH}",
|
||||
"reason": "database_missing"
|
||||
}`), &missing); err != nil {
|
||||
t.Fatalf("decode missing TM summary mirror: %v", err)
|
||||
}
|
||||
if missing.Available || missing.Summary != nil || missing.SchemaVersion != nil ||
|
||||
missing.Reason != "database_missing" {
|
||||
t.Fatalf("missing TM summary=%+v", missing)
|
||||
}
|
||||
}
|
||||
|
||||
+59
-1
@@ -411,6 +411,56 @@ paths:
|
||||
description: Missing or invalid admin token.
|
||||
"503":
|
||||
description: Rust bat translation backend is unavailable.
|
||||
/admin/translation/memory/summary:
|
||||
get:
|
||||
summary: Read Rust-owned Translation Memory summary
|
||||
parameters:
|
||||
- name: translation_memory_path
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Translation Memory availability and candidate/trusted counts.
|
||||
"401":
|
||||
description: Missing or invalid admin token.
|
||||
"503":
|
||||
description: Rust bat Translation Memory backend is unavailable.
|
||||
/admin/translation/memory/query:
|
||||
get:
|
||||
summary: Query Rust-owned Translation Memory records
|
||||
parameters:
|
||||
- name: source_text
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: source_context
|
||||
in: query
|
||||
description: JSON object whose values are strings.
|
||||
schema:
|
||||
type: string
|
||||
- name: limit
|
||||
in: query
|
||||
schema:
|
||||
type: integer
|
||||
format: int64
|
||||
minimum: 1
|
||||
maximum: 1000
|
||||
default: 100
|
||||
- name: translation_memory_path
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Translation Memory matches with reuse decision and provenance.
|
||||
"400":
|
||||
description: Missing source text or invalid context/limit.
|
||||
"401":
|
||||
description: Missing or invalid admin token.
|
||||
"503":
|
||||
description: Rust bat Translation Memory backend is unavailable.
|
||||
/admin/translation/status:
|
||||
get:
|
||||
summary: Read Rust-owned localized release status
|
||||
@@ -430,7 +480,7 @@ paths:
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh, schedule-add, schedule-update, schedule-remove, schedule-run, task-cancel, translation-task-update, translation-worker-run, translation-proofread, localized-publish, localized-rollback]
|
||||
enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh, schedule-add, schedule-update, schedule-remove, schedule-run, task-cancel, translation-task-update, translation-worker-run, translation-proofread, translation-memory-confirm, localized-publish, localized-rollback]
|
||||
requestBody:
|
||||
required: false
|
||||
content:
|
||||
@@ -521,6 +571,14 @@ paths:
|
||||
minimum: 1
|
||||
worker_id:
|
||||
type: string
|
||||
translation_memory_path:
|
||||
type: string
|
||||
record_id:
|
||||
type: string
|
||||
reviewer:
|
||||
type: string
|
||||
reason:
|
||||
type: string
|
||||
translation_file:
|
||||
type: string
|
||||
from_worker:
|
||||
|
||||
@@ -83,6 +83,15 @@ type TranslationBackend interface {
|
||||
TranslationProofread(ctx context.Context) (json.RawMessage, error)
|
||||
}
|
||||
|
||||
// TranslationMemoryBackend exposes the Rust-owned Translation Memory query and
|
||||
// explicit confirmation operations. Go forwards typed requests and responses
|
||||
// but never opens or mutates the TM database itself.
|
||||
type TranslationMemoryBackend interface {
|
||||
TranslationMemorySummary(ctx context.Context, params backendrpc.TranslationMemorySummaryParams) (*backendrpc.TranslationMemorySummaryReport, error)
|
||||
TranslationMemoryQuery(ctx context.Context, params backendrpc.TranslationMemoryQueryParams) (*backendrpc.TranslationMemoryQueryReport, error)
|
||||
TranslationMemoryConfirm(ctx context.Context, params backendrpc.TranslationMemoryConfirmParams) (*backendrpc.TranslationMemoryConfirmReport, error)
|
||||
}
|
||||
|
||||
// LocalizedBackend exposes localized release status and the explicit
|
||||
// publish/rollback controls used by the authenticated dashboard.
|
||||
type LocalizedBackend interface {
|
||||
@@ -182,6 +191,18 @@ func (r RPCClient) TranslationProofread(ctx context.Context) (json.RawMessage, e
|
||||
return r.Client.TranslationProofread(ctx)
|
||||
}
|
||||
|
||||
func (r RPCClient) TranslationMemorySummary(ctx context.Context, params backendrpc.TranslationMemorySummaryParams) (*backendrpc.TranslationMemorySummaryReport, error) {
|
||||
return r.Client.TranslationMemorySummary(ctx, params)
|
||||
}
|
||||
|
||||
func (r RPCClient) TranslationMemoryQuery(ctx context.Context, params backendrpc.TranslationMemoryQueryParams) (*backendrpc.TranslationMemoryQueryReport, error) {
|
||||
return r.Client.TranslationMemoryQuery(ctx, params)
|
||||
}
|
||||
|
||||
func (r RPCClient) TranslationMemoryConfirm(ctx context.Context, params backendrpc.TranslationMemoryConfirmParams) (*backendrpc.TranslationMemoryConfirmReport, error) {
|
||||
return r.Client.TranslationMemoryConfirm(ctx, params)
|
||||
}
|
||||
|
||||
func (r RPCClient) LocalizedStatus(ctx context.Context) (json.RawMessage, error) {
|
||||
return r.Client.LocalizedStatus(ctx)
|
||||
}
|
||||
|
||||
@@ -74,6 +74,8 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("/admin/schedules", s.handleAdminSchedules)
|
||||
mux.HandleFunc("/admin/translation/tasks", s.handleAdminTranslationTasks)
|
||||
mux.HandleFunc("/admin/translation/handoff", s.handleAdminTranslationHandoff)
|
||||
mux.HandleFunc("/admin/translation/memory/summary", s.handleAdminTranslationMemorySummary)
|
||||
mux.HandleFunc("/admin/translation/memory/query", s.handleAdminTranslationMemoryQuery)
|
||||
mux.HandleFunc("/admin/translation/status", s.handleAdminLocalizedStatus)
|
||||
mux.HandleFunc("/admin/control/", s.handleAdminControl)
|
||||
mux.HandleFunc("/admin/", s.handleAdminIndex)
|
||||
@@ -151,6 +153,8 @@ func (s *Server) handleRoot(w http.ResponseWriter, r *http.Request) {
|
||||
"/admin/schedules",
|
||||
"/admin/translation/tasks",
|
||||
"/admin/translation/handoff",
|
||||
"/admin/translation/memory/summary",
|
||||
"/admin/translation/memory/query",
|
||||
"/admin/translation/status",
|
||||
"/admin/control/{action}",
|
||||
},
|
||||
|
||||
+174
-16
@@ -307,26 +307,28 @@ type TranslationTaskListParams struct {
|
||||
// Pointer numeric fields preserve explicit zeroes so Rust can reject invalid
|
||||
// dashboard input instead of receiving omitted defaults.
|
||||
type TranslationWorkerRunParams struct {
|
||||
Provider string `json:"provider,omitempty"`
|
||||
FixturePath string `json:"fixture_path,omitempty"`
|
||||
Concurrency *uint64 `json:"concurrency,omitempty"`
|
||||
MaxAttempts *uint64 `json:"max_attempts,omitempty"`
|
||||
LeaseSeconds *uint64 `json:"lease_seconds,omitempty"`
|
||||
RetryBackoffSeconds *uint64 `json:"retry_backoff_seconds,omitempty"`
|
||||
MaxTasks *uint64 `json:"max_tasks,omitempty"`
|
||||
WorkerID string `json:"worker_id,omitempty"`
|
||||
Provider string `json:"provider,omitempty"`
|
||||
FixturePath string `json:"fixture_path,omitempty"`
|
||||
TranslationMemoryPath string `json:"translation_memory_path,omitempty"`
|
||||
Concurrency *uint64 `json:"concurrency,omitempty"`
|
||||
MaxAttempts *uint64 `json:"max_attempts,omitempty"`
|
||||
LeaseSeconds *uint64 `json:"lease_seconds,omitempty"`
|
||||
RetryBackoffSeconds *uint64 `json:"retry_backoff_seconds,omitempty"`
|
||||
MaxTasks *uint64 `json:"max_tasks,omitempty"`
|
||||
WorkerID string `json:"worker_id,omitempty"`
|
||||
}
|
||||
|
||||
// TranslationWorkerConfig mirrors the accepted worker config returned by Rust.
|
||||
type TranslationWorkerConfig struct {
|
||||
Provider string `json:"provider"`
|
||||
FixturePath string `json:"fixture_path,omitempty"`
|
||||
Concurrency uint64 `json:"concurrency"`
|
||||
MaxAttempts uint64 `json:"max_attempts"`
|
||||
LeaseSeconds uint64 `json:"lease_seconds"`
|
||||
RetryBackoffSeconds uint64 `json:"retry_backoff_seconds"`
|
||||
MaxTasks *uint64 `json:"max_tasks,omitempty"`
|
||||
WorkerID string `json:"worker_id"`
|
||||
Provider string `json:"provider"`
|
||||
FixturePath string `json:"fixture_path,omitempty"`
|
||||
TranslationMemoryPath string `json:"translation_memory_path,omitempty"`
|
||||
Concurrency uint64 `json:"concurrency"`
|
||||
MaxAttempts uint64 `json:"max_attempts"`
|
||||
LeaseSeconds uint64 `json:"lease_seconds"`
|
||||
RetryBackoffSeconds uint64 `json:"retry_backoff_seconds"`
|
||||
MaxTasks *uint64 `json:"max_tasks,omitempty"`
|
||||
WorkerID string `json:"worker_id"`
|
||||
}
|
||||
|
||||
// TranslationWorkerRunResult is returned when translation.worker.run is queued.
|
||||
@@ -336,6 +338,144 @@ type TranslationWorkerRunResult struct {
|
||||
Worker TranslationWorkerConfig `json:"worker"`
|
||||
}
|
||||
|
||||
// TranslationMemoryContext is the stable TextUnit context sent to Rust.
|
||||
type TranslationMemoryContext map[string]string
|
||||
|
||||
// TranslationMemorySourceKind identifies who supplied the translation.
|
||||
type TranslationMemorySourceKind string
|
||||
|
||||
const (
|
||||
TranslationMemorySourceProvider TranslationMemorySourceKind = "provider"
|
||||
TranslationMemorySourceManual TranslationMemorySourceKind = "manual"
|
||||
TranslationMemorySourceImported TranslationMemorySourceKind = "imported"
|
||||
)
|
||||
|
||||
// TranslationMemoryTrustStatus is the Rust-owned review state.
|
||||
type TranslationMemoryTrustStatus string
|
||||
|
||||
const (
|
||||
TranslationMemoryStatusCandidate TranslationMemoryTrustStatus = "candidate"
|
||||
TranslationMemoryStatusTrusted TranslationMemoryTrustStatus = "trusted"
|
||||
TranslationMemoryStatusSuperseded TranslationMemoryTrustStatus = "superseded"
|
||||
TranslationMemoryStatusRejected TranslationMemoryTrustStatus = "rejected"
|
||||
)
|
||||
|
||||
// TranslationMemoryMatchKind describes why a record was returned.
|
||||
type TranslationMemoryMatchKind string
|
||||
|
||||
const (
|
||||
TranslationMemoryMatchStrongExact TranslationMemoryMatchKind = "strong_exact"
|
||||
TranslationMemoryMatchCandidateExact TranslationMemoryMatchKind = "candidate_exact"
|
||||
TranslationMemoryMatchSourceOnly TranslationMemoryMatchKind = "source_only"
|
||||
)
|
||||
|
||||
// TranslationMemorySummaryParams selects an optional Rust-owned TM database.
|
||||
type TranslationMemorySummaryParams struct {
|
||||
TranslationMemoryPath string `json:"translation_memory_path,omitempty"`
|
||||
}
|
||||
|
||||
// TranslationMemoryQueryParams queries Rust-owned TM records by raw source and
|
||||
// optional complete TextUnit context.
|
||||
type TranslationMemoryQueryParams struct {
|
||||
TranslationMemoryPath string `json:"translation_memory_path,omitempty"`
|
||||
SourceText string `json:"source_text"`
|
||||
SourceContext TranslationMemoryContext `json:"source_context,omitempty"`
|
||||
Limit *uint64 `json:"limit,omitempty"`
|
||||
}
|
||||
|
||||
// TranslationMemoryConfirmParams explicitly promotes one candidate record.
|
||||
type TranslationMemoryConfirmParams struct {
|
||||
TranslationMemoryPath string `json:"translation_memory_path,omitempty"`
|
||||
RecordID string `json:"record_id"`
|
||||
Reviewer string `json:"reviewer"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// TranslationMemorySummary mirrors translation.memory.summary data.
|
||||
type TranslationMemorySummary struct {
|
||||
SchemaVersion uint64 `json:"schema_version"`
|
||||
RecordCount uint64 `json:"record_count"`
|
||||
TrustedCount uint64 `json:"trusted_count"`
|
||||
CandidateCount uint64 `json:"candidate_count"`
|
||||
SupersededCount uint64 `json:"superseded_count"`
|
||||
RejectedCount uint64 `json:"rejected_count"`
|
||||
}
|
||||
|
||||
// TranslationMemorySummaryReport distinguishes a missing database from an
|
||||
// available database with an empty summary.
|
||||
type TranslationMemorySummaryReport struct {
|
||||
Available bool `json:"available"`
|
||||
Path string `json:"path"`
|
||||
SchemaVersion *uint64 `json:"schema_version,omitempty"`
|
||||
Summary *TranslationMemorySummary `json:"summary,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// TranslationMemorySourceTrace mirrors the Rust TextUnit/provider provenance.
|
||||
type TranslationMemorySourceTrace struct {
|
||||
OfficialReleaseID string `json:"official_release_id"`
|
||||
UnitID *string `json:"unit_id,omitempty"`
|
||||
TaskID *string `json:"task_id,omitempty"`
|
||||
Destination *string `json:"destination,omitempty"`
|
||||
ArchiveEntry *string `json:"archive_entry,omitempty"`
|
||||
SerializedFile *string `json:"serialized_file,omitempty"`
|
||||
PathID *int64 `json:"path_id,omitempty"`
|
||||
ClassID *int32 `json:"class_id,omitempty"`
|
||||
FieldPath *string `json:"field_path,omitempty"`
|
||||
Format *string `json:"format,omitempty"`
|
||||
AssetName *string `json:"asset_name,omitempty"`
|
||||
TextSourceKind *string `json:"text_source_kind,omitempty"`
|
||||
SourceURL *string `json:"source_url,omitempty"`
|
||||
}
|
||||
|
||||
// TranslationMemoryEntry mirrors a persisted Rust TM record.
|
||||
type TranslationMemoryEntry struct {
|
||||
RecordID string `json:"record_id"`
|
||||
SourceText string `json:"source_text"`
|
||||
SourceHash string `json:"source_hash"`
|
||||
NormalizedSourceText string `json:"normalized_source_text"`
|
||||
SourceContext TranslationMemoryContext `json:"source_context,omitempty"`
|
||||
SourceContextHash string `json:"source_context_hash"`
|
||||
TranslatedText string `json:"translated_text"`
|
||||
TranslationSourceKind TranslationMemorySourceKind `json:"translation_source_kind"`
|
||||
TrustStatus TranslationMemoryTrustStatus `json:"trust_status"`
|
||||
OfficialReleaseID string `json:"official_release_id"`
|
||||
SourceTrace TranslationMemorySourceTrace `json:"source_trace"`
|
||||
Provider *string `json:"provider,omitempty"`
|
||||
ProviderRunID *string `json:"provider_run_id,omitempty"`
|
||||
CreatedUnixSeconds uint64 `json:"created_unix_seconds"`
|
||||
UpdatedUnixSeconds uint64 `json:"updated_unix_seconds"`
|
||||
TrustedUnixSeconds *uint64 `json:"trusted_unix_seconds,omitempty"`
|
||||
TrustedBy *string `json:"trusted_by,omitempty"`
|
||||
TrustedReason *string `json:"trusted_reason,omitempty"`
|
||||
SupersedesRecordID *string `json:"supersedes_record_id,omitempty"`
|
||||
SupersededByRecordID *string `json:"superseded_by_record_id,omitempty"`
|
||||
}
|
||||
|
||||
// TranslationMemoryMatch is one Rust-selected match and its reuse decision.
|
||||
type TranslationMemoryMatch struct {
|
||||
Entry TranslationMemoryEntry `json:"entry"`
|
||||
MatchKind TranslationMemoryMatchKind `json:"match_kind"`
|
||||
CanAutoReuse bool `json:"can_auto_reuse"`
|
||||
}
|
||||
|
||||
// TranslationMemoryQueryReport mirrors translation.memory.query data.
|
||||
type TranslationMemoryQueryReport struct {
|
||||
Available bool `json:"available"`
|
||||
Path string `json:"path"`
|
||||
SourceText string `json:"source_text"`
|
||||
SourceContext TranslationMemoryContext `json:"source_context,omitempty"`
|
||||
Matches []TranslationMemoryMatch `json:"matches"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// TranslationMemoryConfirmReport mirrors translation.memory.confirm data.
|
||||
type TranslationMemoryConfirmReport struct {
|
||||
Available bool `json:"available"`
|
||||
Path string `json:"path"`
|
||||
Entry TranslationMemoryEntry `json:"entry"`
|
||||
}
|
||||
|
||||
// LocalizedPublishParams selects the source of one localized release
|
||||
// publication. TranslationFile and FromWorker are mutually exclusive.
|
||||
type LocalizedPublishParams struct {
|
||||
@@ -593,6 +733,24 @@ func (c *Client) TranslationWorkerRun(ctx context.Context, params TranslationWor
|
||||
return &out, err
|
||||
}
|
||||
|
||||
func (c *Client) TranslationMemorySummary(ctx context.Context, params TranslationMemorySummaryParams) (*TranslationMemorySummaryReport, error) {
|
||||
var out TranslationMemorySummaryReport
|
||||
_, err := c.Call(ctx, "translation.memory.summary", params, &out)
|
||||
return &out, err
|
||||
}
|
||||
|
||||
func (c *Client) TranslationMemoryQuery(ctx context.Context, params TranslationMemoryQueryParams) (*TranslationMemoryQueryReport, error) {
|
||||
var out TranslationMemoryQueryReport
|
||||
_, err := c.Call(ctx, "translation.memory.query", params, &out)
|
||||
return &out, err
|
||||
}
|
||||
|
||||
func (c *Client) TranslationMemoryConfirm(ctx context.Context, params TranslationMemoryConfirmParams) (*TranslationMemoryConfirmReport, error) {
|
||||
var out TranslationMemoryConfirmReport
|
||||
_, err := c.Call(ctx, "translation.memory.confirm", params, &out)
|
||||
return &out, err
|
||||
}
|
||||
|
||||
func (c *Client) LocalizedPublish(ctx context.Context, params LocalizedPublishParams) (json.RawMessage, error) {
|
||||
return c.rawData(ctx, "localized.publish", params)
|
||||
}
|
||||
|
||||
@@ -489,6 +489,7 @@ func TestTranslationWorkerRunSendsProviderConfig(t *testing.T) {
|
||||
}
|
||||
if params.Provider != "mock" ||
|
||||
params.FixturePath != "/tmp/mock-provider.json" ||
|
||||
params.TranslationMemoryPath != "/tmp/translation-memory.sqlite" ||
|
||||
params.Concurrency == nil || *params.Concurrency != concurrency ||
|
||||
params.MaxAttempts == nil || *params.MaxAttempts != maxAttempts ||
|
||||
params.LeaseSeconds == nil || *params.LeaseSeconds != leaseSeconds ||
|
||||
@@ -506,14 +507,15 @@ func TestTranslationWorkerRunSendsProviderConfig(t *testing.T) {
|
||||
"task_id": "task-worker-1",
|
||||
"kind": "translation.worker.run",
|
||||
"worker": map[string]any{
|
||||
"provider": "mock",
|
||||
"fixture_path": "/tmp/mock-provider.json",
|
||||
"concurrency": concurrency,
|
||||
"max_attempts": maxAttempts,
|
||||
"lease_seconds": leaseSeconds,
|
||||
"retry_backoff_seconds": retryBackoff,
|
||||
"max_tasks": maxTasks,
|
||||
"worker_id": "dashboard-worker",
|
||||
"provider": "mock",
|
||||
"fixture_path": "/tmp/mock-provider.json",
|
||||
"translation_memory_path": "/tmp/translation-memory.sqlite",
|
||||
"concurrency": concurrency,
|
||||
"max_attempts": maxAttempts,
|
||||
"lease_seconds": leaseSeconds,
|
||||
"retry_backoff_seconds": retryBackoff,
|
||||
"max_tasks": maxTasks,
|
||||
"worker_id": "dashboard-worker",
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -521,14 +523,15 @@ func TestTranslationWorkerRunSendsProviderConfig(t *testing.T) {
|
||||
})
|
||||
|
||||
result, err := client.TranslationWorkerRun(context.Background(), TranslationWorkerRunParams{
|
||||
Provider: "mock",
|
||||
FixturePath: "/tmp/mock-provider.json",
|
||||
Concurrency: &concurrency,
|
||||
MaxAttempts: &maxAttempts,
|
||||
LeaseSeconds: &leaseSeconds,
|
||||
RetryBackoffSeconds: &retryBackoff,
|
||||
MaxTasks: &maxTasks,
|
||||
WorkerID: "dashboard-worker",
|
||||
Provider: "mock",
|
||||
FixturePath: "/tmp/mock-provider.json",
|
||||
TranslationMemoryPath: "/tmp/translation-memory.sqlite",
|
||||
Concurrency: &concurrency,
|
||||
MaxAttempts: &maxAttempts,
|
||||
LeaseSeconds: &leaseSeconds,
|
||||
RetryBackoffSeconds: &retryBackoff,
|
||||
MaxTasks: &maxTasks,
|
||||
WorkerID: "dashboard-worker",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("TranslationWorkerRun error: %v", err)
|
||||
@@ -536,12 +539,192 @@ func TestTranslationWorkerRunSendsProviderConfig(t *testing.T) {
|
||||
if result.TaskID != "task-worker-1" ||
|
||||
result.Kind != "translation.worker.run" ||
|
||||
result.Worker.Concurrency != concurrency ||
|
||||
result.Worker.TranslationMemoryPath != "/tmp/translation-memory.sqlite" ||
|
||||
result.Worker.RetryBackoffSeconds != retryBackoff ||
|
||||
result.Worker.MaxTasks == nil || *result.Worker.MaxTasks != maxTasks {
|
||||
t.Fatalf("unexpected result: %#v", result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTranslationMemoryTypedContract(t *testing.T) {
|
||||
limit := uint64(25)
|
||||
client := newTestClient(t, func(t *testing.T, req testRequest) testResponse {
|
||||
switch req.Method {
|
||||
case "translation.memory.summary":
|
||||
var params TranslationMemorySummaryParams
|
||||
if err := json.Unmarshal(req.Params, ¶ms); err != nil {
|
||||
t.Fatalf("decode summary params: %v", err)
|
||||
}
|
||||
if params.TranslationMemoryPath != "/var/lib/bat/translation-memory.sqlite" {
|
||||
t.Fatalf("summary params=%#v", params)
|
||||
}
|
||||
return testResponse{
|
||||
Result: testEnvelope{
|
||||
OK: true, Status: "ok", RequestID: "req-tm-summary",
|
||||
Data: map[string]any{
|
||||
"available": true,
|
||||
"path": "/var/lib/bat/translation-memory.sqlite",
|
||||
"schema_version": 1,
|
||||
"summary": map[string]any{
|
||||
"schema_version": 1,
|
||||
"record_count": 3,
|
||||
"trusted_count": 1,
|
||||
"candidate_count": 1,
|
||||
"superseded_count": 1,
|
||||
"rejected_count": 0,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
case "translation.memory.query":
|
||||
var params TranslationMemoryQueryParams
|
||||
if err := json.Unmarshal(req.Params, ¶ms); err != nil {
|
||||
t.Fatalf("decode query params: %v", err)
|
||||
}
|
||||
if params.SourceText != "Hello" ||
|
||||
params.SourceContext["destination"] != "Bundle/dialogue.bundle" ||
|
||||
params.Limit == nil || *params.Limit != limit {
|
||||
t.Fatalf("query params=%#v", params)
|
||||
}
|
||||
return testResponse{
|
||||
Result: testEnvelope{
|
||||
OK: true, Status: "ok", RequestID: "req-tm-query",
|
||||
Data: map[string]any{
|
||||
"available": true,
|
||||
"path": "/var/lib/bat/translation-memory.sqlite",
|
||||
"source_text": "Hello",
|
||||
"source_context": map[string]any{
|
||||
"destination": "Bundle/dialogue.bundle",
|
||||
"field_path": "Dialog.Message",
|
||||
},
|
||||
"matches": []any{
|
||||
map[string]any{
|
||||
"entry": map[string]any{
|
||||
"record_id": "tm-record-1",
|
||||
"source_text": "Hello",
|
||||
"source_hash": "hash-source",
|
||||
"normalized_source_text": "hello",
|
||||
"source_context": map[string]any{
|
||||
"destination": "Bundle/dialogue.bundle",
|
||||
"field_path": "Dialog.Message",
|
||||
},
|
||||
"source_context_hash": "hash-context",
|
||||
"translated_text": "你好",
|
||||
"translation_source_kind": "provider",
|
||||
"trust_status": "trusted",
|
||||
"official_release_id": "release-1",
|
||||
"source_trace": map[string]any{
|
||||
"official_release_id": "release-1",
|
||||
"unit_id": "textunit-1",
|
||||
"task_id": "task-1",
|
||||
"destination": "Bundle/dialogue.bundle",
|
||||
"archive_entry": "dialogue.json",
|
||||
"serialized_file": "globalgamemanagers",
|
||||
"path_id": 42,
|
||||
"class_id": 114,
|
||||
"field_path": "Dialog.Message",
|
||||
"format": "json",
|
||||
"asset_name": "Dialogue",
|
||||
"text_source_kind": "text_asset",
|
||||
},
|
||||
"provider": "mock",
|
||||
"provider_run_id": "provider-run-1",
|
||||
"created_unix_seconds": 100,
|
||||
"updated_unix_seconds": 200,
|
||||
"trusted_unix_seconds": 200,
|
||||
"trusted_by": "reviewer",
|
||||
"trusted_reason": "reviewed",
|
||||
},
|
||||
"match_kind": "strong_exact",
|
||||
"can_auto_reuse": true,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
case "translation.memory.confirm":
|
||||
var params TranslationMemoryConfirmParams
|
||||
if err := json.Unmarshal(req.Params, ¶ms); err != nil {
|
||||
t.Fatalf("decode confirm params: %v", err)
|
||||
}
|
||||
if params.RecordID != "tm-record-1" ||
|
||||
params.Reviewer != "reviewer" ||
|
||||
params.Reason != "reviewed" ||
|
||||
params.TranslationMemoryPath != "/var/lib/bat/translation-memory.sqlite" {
|
||||
t.Fatalf("confirm params=%#v", params)
|
||||
}
|
||||
return testResponse{
|
||||
Result: testEnvelope{
|
||||
OK: true, Status: "ok", RequestID: "req-tm-confirm",
|
||||
Data: map[string]any{
|
||||
"available": true,
|
||||
"path": "/var/lib/bat/translation-memory.sqlite",
|
||||
"entry": map[string]any{
|
||||
"record_id": "tm-record-1",
|
||||
"translation_source_kind": "provider",
|
||||
"trust_status": "trusted",
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
default:
|
||||
t.Fatalf("unexpected method %q", req.Method)
|
||||
return testResponse{}
|
||||
}
|
||||
})
|
||||
|
||||
summary, err := client.TranslationMemorySummary(context.Background(), TranslationMemorySummaryParams{
|
||||
TranslationMemoryPath: "/var/lib/bat/translation-memory.sqlite",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("TranslationMemorySummary error: %v", err)
|
||||
}
|
||||
if !summary.Available || summary.Summary == nil ||
|
||||
summary.Summary.TrustedCount != 1 || summary.Summary.CandidateCount != 1 {
|
||||
t.Fatalf("summary=%#v", summary)
|
||||
}
|
||||
|
||||
query, err := client.TranslationMemoryQuery(context.Background(), TranslationMemoryQueryParams{
|
||||
TranslationMemoryPath: "/var/lib/bat/translation-memory.sqlite",
|
||||
SourceText: "Hello",
|
||||
SourceContext: TranslationMemoryContext{
|
||||
"destination": "Bundle/dialogue.bundle",
|
||||
"field_path": "Dialog.Message",
|
||||
},
|
||||
Limit: &limit,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("TranslationMemoryQuery error: %v", err)
|
||||
}
|
||||
if len(query.Matches) != 1 ||
|
||||
query.Matches[0].MatchKind != "strong_exact" ||
|
||||
!query.Matches[0].CanAutoReuse ||
|
||||
query.Matches[0].Entry.TrustStatus != "trusted" ||
|
||||
query.Matches[0].Entry.TranslationSourceKind != "provider" ||
|
||||
query.Matches[0].Entry.SourceTrace.UnitID == nil ||
|
||||
*query.Matches[0].Entry.SourceTrace.UnitID != "textunit-1" ||
|
||||
query.Matches[0].Entry.SourceTrace.PathID == nil ||
|
||||
*query.Matches[0].Entry.SourceTrace.PathID != 42 ||
|
||||
query.Matches[0].Entry.TrustedBy == nil ||
|
||||
*query.Matches[0].Entry.TrustedBy != "reviewer" {
|
||||
t.Fatalf("query=%#v", query)
|
||||
}
|
||||
|
||||
confirmed, err := client.TranslationMemoryConfirm(context.Background(), TranslationMemoryConfirmParams{
|
||||
TranslationMemoryPath: "/var/lib/bat/translation-memory.sqlite",
|
||||
RecordID: "tm-record-1",
|
||||
Reviewer: "reviewer",
|
||||
Reason: "reviewed",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("TranslationMemoryConfirm error: %v", err)
|
||||
}
|
||||
if !confirmed.Available || confirmed.Entry.TrustStatus != "trusted" ||
|
||||
confirmed.Entry.RecordID != "tm-record-1" {
|
||||
t.Fatalf("confirmed=%#v", confirmed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTranslationProofreadUsesRustMethod(t *testing.T) {
|
||||
client := newTestClient(t, func(t *testing.T, req testRequest) testResponse {
|
||||
if req.Method != "translation.proofread" {
|
||||
|
||||
Reference in New Issue
Block a user