mirror of
https://github.com/Yuyi-Oak/BlueArchiveToolkit.git
synced 2026-09-18 06:34:54 +08:00
fix: 完成下载并发与翻译交接链路
This commit is contained in:
+7
-2
@@ -20,7 +20,7 @@ Rust 侧官方日服资源链路已经从实验验证推进到正式入口:
|
|||||||
3. 默认平台为 `Windows + Android`。
|
3. 默认平台为 `Windows + Android`。
|
||||||
4. 能生成官方全量 pull plan,执行真实下载,维护 release 内的 `official-download-manifest.json`。
|
4. 能生成官方全量 pull plan,执行真实下载,维护 release 内的 `official-download-manifest.json`。
|
||||||
5. 下载后使用本地 manifest 的 size + BLAKE3 校验复用文件;所有 `.zip` 在下载验收、复用、本地 audit/verify 时做 ZIP 结构校验;官方 seed `.hash` 使用标准 `xxHash32(seed=0)` 强校验(早期实现的非标准 avalanche 常量已修正)。
|
5. 下载后使用本地 manifest 的 size + BLAKE3 校验复用文件;所有 `.zip` 在下载验收、复用、本地 audit/verify 时做 ZIP 结构校验;官方 seed `.hash` 使用标准 `xxHash32(seed=0)` 强校验(早期实现的非标准 avalanche 常量已修正)。
|
||||||
6. 支持 `.part` 断点续传、失败后 clean retry、本地 manifest audit/repair、失败 staging 恢复复用、403/404/5xx 分类重试(重试带指数退避)、下载 quarantine 诊断,以及旧 launcher 包官方 primary/backup CDN 切换。启动器/server-info 先行更新但 client-patch seed marker 或必需 seed catalog 尚未开放时,会进入 `waiting_for_official_resources`,保留现有 `current`,不创建失败 staging,也不写入失败版本循环;启用 `--auto-discover` 的非 dry-run 会写入 `<output>/official-launcher-bootstrap.pending.json` 作为维护期证据。下载执行保持顺序处理;manifest/quarantine 簿记与 seed `.hash` 校验仍逐项执行,`fail-fast` 与「不发布不完整资源」不变量不变。下载进度按已完成数量单调上报,不再使用 plan 序号计算百分比。
|
6. 支持 `.part` 断点续传、失败后 clean retry、本地 manifest audit/repair、失败 staging 恢复复用、403/404/5xx 分类重试(重试带指数退避)、下载 quarantine 诊断,以及旧 launcher 包官方 primary/backup CDN 切换。启动器/server-info 先行更新但 client-patch seed marker 或必需 seed catalog 尚未开放时,会进入 `waiting_for_official_resources`,保留现有 `current`,不创建失败 staging,也不写入失败版本循环;启用 `--auto-discover` 的非 dry-run 会写入 `<output>/official-launcher-bootstrap.pending.json` 作为维护期证据。下载默认使用 8 个独立 worker,范围为 `1..=256`;每个 worker 完成当前 URL 后立即从共享计划队列领取下一个任务,进度按实际完成顺序即时上报,最终 report 资源列表仍按计划顺序输出。manifest/quarantine 簿记与 seed `.hash` 校验仍逐项执行,`fail-fast` 与「不发布不完整资源」不变量不变。下载进度按已完成数量单调上报,不再使用 plan 序号计算百分比。
|
||||||
7. 支持 curl 传输层本地代理:默认自动检测 `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY` 及小写环境变量(带凭据的代理推荐用环境变量配置),也可用 `--proxy <URL>` 显式指定或 `--no-proxy` 强制直连;代理决策会写入 progress log、daemon log 和 `bat doctor` 诊断输出。代理凭据不落世界可读位置:日志/`status` 脱敏,传给 curl 经 `ALL_PROXY` 环境变量而非 argv,`--daemon` 下经环境变量下传后台子进程、不进子进程 argv 或 `bat-status.json`,复用凭据存于 `bat-proxy.secret`(`0600`)且 `clean-stable` 会清除。
|
7. 支持 curl 传输层本地代理:默认自动检测 `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY` 及小写环境变量(带凭据的代理推荐用环境变量配置),也可用 `--proxy <URL>` 显式指定或 `--no-proxy` 强制直连;代理决策会写入 progress log、daemon log 和 `bat doctor` 诊断输出。代理凭据不落世界可读位置:日志/`status` 脱敏,传给 curl 经 `ALL_PROXY` 环境变量而非 argv,`--daemon` 下经环境变量下传后台子进程、不进子进程 argv 或 `bat-status.json`,复用凭据存于 `bat-proxy.secret`(`0600`)且 `clean-stable` 会清除。
|
||||||
8. `bat --watch` 可常驻运行,`bat --daemon` 可后台运行并用 `bat status` / `bat stop` / `bat restart` / `bat reload` / `bat logs` 管理;daemon 使用 `bat.sock` Unix socket JSON-RPC 作为 live 控制通道,PID/状态/日志文件作为快照和 fallback,`bat-events.jsonl` 记录带轮转的结构化事件日志,`bat-control.lock` 串行化控制命令;正常检查默认每 1 小时一次;远端和本地一致时默认静默,失败后默认 60 秒快速重试,官方资源端尚未开放时状态为 `waiting` 并同样按错误重试间隔探测;`resource.state` / `catalog.status` / `parse.status` / `localized.status` 会返回 `status` 与稳定 `status_code`(如 `official.up_to_date`、`official.published`、`parse.completed`、`translation.queued_offline`、`localized.published`、`distribution.ready`),供 `bat-api` 等读侧判断阶段、终态和重试属性;CLI 默认向 stdout 输出人类可读摘要,向 stderr 输出 ASCII banner、progress log、失败分类和 quarantine 状态,需要机器输出时使用 `--json --no-progress`。
|
8. `bat --watch` 可常驻运行,`bat --daemon` 可后台运行并用 `bat status` / `bat stop` / `bat restart` / `bat reload` / `bat logs` 管理;daemon 使用 `bat.sock` Unix socket JSON-RPC 作为 live 控制通道,PID/状态/日志文件作为快照和 fallback,`bat-events.jsonl` 记录带轮转的结构化事件日志,`bat-control.lock` 串行化控制命令;正常检查默认每 1 小时一次;远端和本地一致时默认静默,失败后默认 60 秒快速重试,官方资源端尚未开放时状态为 `waiting` 并同样按错误重试间隔探测;`resource.state` / `catalog.status` / `parse.status` / `localized.status` 会返回 `status` 与稳定 `status_code`(如 `official.up_to_date`、`official.published`、`parse.completed`、`translation.queued_offline`、`localized.published`、`distribution.ready`),供 `bat-api` 等读侧判断阶段、终态和重试属性;CLI 默认向 stdout 输出人类可读摘要,向 stderr 输出 ASCII banner、progress log、失败分类和 quarantine 状态,需要机器输出时使用 `--json --no-progress`。
|
||||||
9. 远端 snapshot 未变化但输出目录为空时,会按首次运行执行全量拉取;官方 seed `.hash` 校验失败时会清理对应 manifest 条目,避免失败产物被后续本地 audit 误判为可复用。
|
9. 远端 snapshot 未变化但输出目录为空时,会按首次运行执行全量拉取;官方 seed `.hash` 校验失败时会清理对应 manifest 条目,避免失败产物被后续本地 audit 误判为可复用。
|
||||||
@@ -32,6 +32,11 @@ Rust 侧官方日服资源链路已经从实验验证推进到正式入口:
|
|||||||
|
|
||||||
仍需明确:这不是完整产品完成。完整 AssetBundle 重打包、翻译、Web、以及 `bat-api` 的服务器联调/可选业务扩展仍是后续工作;G-008(产品级 Go 同步 CLI)已决策关闭。真实官方网络全量拉取 smoke 已固化(G-018 已关闭);真实大文件与运行报告默认在 `/tmp` 隔离目录,不纳入 Git。Go 细节见 `docs/reports/GO_STATUS.md`。
|
仍需明确:这不是完整产品完成。完整 AssetBundle 重打包、翻译、Web、以及 `bat-api` 的服务器联调/可选业务扩展仍是后续工作;G-008(产品级 Go 同步 CLI)已决策关闭。真实官方网络全量拉取 smoke 已固化(G-018 已关闭);真实大文件与运行报告默认在 `/tmp` 隔离目录,不纳入 Git。Go 细节见 `docs/reports/GO_STATUS.md`。
|
||||||
|
|
||||||
|
当前翻译交接还包括 `translation-tasks.sqlite` 和版本化 `translation-handoff.json`;
|
||||||
|
`translation.tasks` 查询单项 worker 状态,`translation.handoff` 查询完整
|
||||||
|
job/unit/provider run 状态。当前下载实现使用默认 8 个独立 worker,完成后动态领取
|
||||||
|
任务,最终资源报告按 pull plan 顺序输出。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. 权威文档入口
|
## 2. 权威文档入口
|
||||||
@@ -281,7 +286,7 @@ cargo run -p bat-infrastructure --bin bat -- \
|
|||||||
|
|
||||||
## 6. 当前阻塞项
|
## 6. 当前阻塞项
|
||||||
|
|
||||||
GitHub issue 状态:#1 已关闭;#17 已按 wontfix 关闭(多线程下载入口已移除,下载回归顺序执行并保留指数退避与单调进度上报,子 issue #20–#23 均已关闭)。其他 open issue 的实时标签以 GitHub 为准。
|
GitHub issue 状态:#1 已关闭;#17 的历史决定不代表当前下载实现,现行默认并发为 8,范围 `1..=256`,每个独立 worker 完成后立即领取下一个任务,进度按完成事件即时统计并保持 report 计划顺序;子 issue #20–#23 均已关闭。其他 open issue 的实时标签以 GitHub 为准。
|
||||||
|
|
||||||
下一阶段必须优先完成:
|
下一阶段必须优先完成:
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -174,7 +174,7 @@ BlueArchiveToolkit 不是一次性脚本,也不是演示项目。最终交付
|
|||||||
|
|
||||||
1. Addressables Catalog 真实字段解析:**部分完成**。当前已覆盖 path、hash、size、address、dependencies、metadata 和真实形态 fixture/golden;仍需继续覆盖更多官方 catalog 结构变体。
|
1. Addressables Catalog 真实字段解析:**部分完成**。当前已覆盖 path、hash、size、address、dependencies、metadata 和真实形态 fixture/golden;仍需继续覆盖更多官方 catalog 结构变体。
|
||||||
2. 资源版本、区域、渠道、远端 URL、Hash、大小、依赖关系模型:**部分完成**。`Resource` 和官方 endpoint/snapshot 模型已扩展;仍需冻结 Go CLI/API 可见模型。
|
2. 资源版本、区域、渠道、远端 URL、Hash、大小、依赖关系模型:**部分完成**。`Resource` 和官方 endpoint/snapshot 模型已扩展;仍需冻结 Go CLI/API 可见模型。
|
||||||
3. Rust 官方下载器:**已完成当前生产入口需要的核心能力**。包含官方 URL 校验、`.part` 续传、重试、本地 manifest size+BLAKE3 校验、官方 seed `.hash` 校验和 repair。
|
3. Rust 官方下载器:**已完成当前生产入口需要的核心能力**。包含官方 URL 校验、`.part` 续传、重试、本地 manifest size+BLAKE3 校验、官方 seed `.hash` 校验、repair,以及默认 8、范围 `1..=256` 的有界并发 scheduler;worker 动态领取任务,进度按完成数单调上报,report 保持 plan 顺序。
|
||||||
4. Rust 自动更新入口:**已完成当前生产入口**。`bat` 支持 snapshot、marker diff、bootstrap cache、one-shot、`--watch`、`--daemon`、默认 1 小时间隔、北京时间固定强制刷新,以及 Unix socket JSON-RPC 后台运维命令返回。
|
4. Rust 自动更新入口:**已完成当前生产入口**。`bat` 支持 snapshot、marker diff、bootstrap cache、one-shot、`--watch`、`--daemon`、默认 1 小时间隔、北京时间固定强制刷新,以及 Unix socket JSON-RPC 后台运维命令返回。
|
||||||
5. Go 入口边界:**已冻结**。同步命令行 = Rust `bat`(G-008 关闭);资源分发 = `bat-api` MVP(G-009 部分完成)。详见 `docs/reports/GO_STATUS.md`。
|
5. Go 入口边界:**已冻结**。同步命令行 = Rust `bat`(G-008 关闭);资源分发 = `bat-api` MVP(G-009 部分完成)。详见 `docs/reports/GO_STATUS.md`。
|
||||||
6. 用户级 `sync`、`manifest inspect`、`cache status`:**未完成**。Rust `bat --json` 是当前稳定进程边界;`bat-ffi` 只提供可选兼容用的 Manifest inspect 和 sync plan JSON helper。
|
6. 用户级 `sync`、`manifest inspect`、`cache status`:**未完成**。Rust `bat --json` 是当前稳定进程边界;`bat-ffi` 只提供可选兼容用的 Manifest inspect 和 sync plan JSON helper。
|
||||||
|
|||||||
@@ -13,13 +13,13 @@
|
|||||||
- `bat-adapters` Unity、Manifest、Client 集成框架,以及当前真实形态 Addressables catalog 解析覆盖,含 `m_Crc` 提取和 UnityFS 解包/TextAsset 提取基础校验。
|
- `bat-adapters` Unity、Manifest、Client 集成框架,以及当前真实形态 Addressables catalog 解析覆盖,含 `m_Crc` 提取和 UnityFS 解包/TextAsset 提取基础校验。
|
||||||
- `bat-cas-engine` CAS V1:原子写入、BLAKE3 校验、引用计数、GC、并发写入测试、损坏检测。
|
- `bat-cas-engine` CAS V1:原子写入、BLAKE3 校验、引用计数、GC、并发写入测试、损坏检测。
|
||||||
- `bat-infrastructure` CAS 适配层、SQLite Resource Repository、资源导入服务、官方资源 pull/update 服务。
|
- `bat-infrastructure` CAS 适配层、SQLite Resource Repository、资源导入服务、官方资源 pull/update 服务。
|
||||||
- `bat`:官方资源自动发现、全量拉取、原子发布到 `current -> versions/<id>`、本地 manifest audit/repair、`.part` 断点续传、403/404/5xx 分类重试、指数退避、顺序下载、下载 quarantine 诊断、ZIP 结构校验、官方 seed `.hash` 校验、snapshot/cache、版本化 `official-launcher-bootstrap.json`、`--watch` 常驻更新、`--daemon` 后台运行,以及 Unix socket JSON-RPC live control/backend 方法(`daemon.status/logs/stop/restart/reload/refresh/doctor`、`resource.sync/verify/repair/state/manifest/list/index`、`parse.status/text_units/errors`、`localized.status`、`catalog.*`、`task.*`)。
|
- `bat`:官方资源自动发现、全量拉取、原子发布到 `current -> versions/<id>`、本地 manifest audit/repair、`.part` 断点续传、403/404/5xx 分类重试、指数退避、默认并发 8(可配置 `1..=256`,report 按 plan 顺序、进度按完成数单调上报)、下载 quarantine 诊断、ZIP 结构校验、官方 seed `.hash` 校验、snapshot/cache、版本化 `official-launcher-bootstrap.json`、`--watch` 常驻更新、`--daemon` 后台运行,以及 Unix socket JSON-RPC live control/backend 方法(`daemon.status/logs/stop/restart/reload/refresh/doctor`、`resource.sync/verify/repair/state/manifest/list/index`、`parse.status/text_units/errors`、`localized.status`、`catalog.*`、`task.*`)。
|
||||||
- `internal/backendrpc`:Go 侧 typed Unix socket JSON-RPC client,是 `bat-api` 调用 Rust daemon 的默认路径。
|
- `internal/backendrpc`:Go 侧 typed Unix socket JSON-RPC client,是 `bat-api` 调用 Rust daemon 的默认路径。
|
||||||
- `cmd/bat-api`:资源 bootstrap + 分发 HTTP MVP(issue #19 / G-009);`/v1/bootstrap` 和 `/v1/launcher/bootstrap` 组织 `bat` 已发布 release 的启动前资源入口,launcher 形状兼容端点仅输出资源 metadata / GameMainConfig 引导,`/healthz` 暴露 RPC refresh 诊断,`/readyz` 做 release readiness,CDN path 支持 `GET`/`HEAD`/`Range`、ETag、Last-Modified 和缓存头;玩家-facing 控制面已具备 token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI 和管理控制白名单;`.env` 配置端口/RPC socket/刷新周期;生产资源根来自 RPC,不负责自动拉取。
|
- `cmd/bat-api`:资源 bootstrap + 分发 HTTP MVP(issue #19 / G-009);`/v1/bootstrap` 和 `/v1/launcher/bootstrap` 组织 `bat` 已发布 release 的启动前资源入口,launcher 形状兼容端点仅输出资源 metadata / GameMainConfig 引导,`/healthz` 暴露 RPC refresh 诊断,`/readyz` 做 release readiness,CDN path 支持 `GET`/`HEAD`/`Range`、ETag、Last-Modified 和缓存头;玩家-facing 控制面已具备 token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI 和管理控制白名单;`.env` 配置端口/RPC socket/刷新周期;生产资源根来自 RPC,不负责自动拉取。
|
||||||
- Go 边界权威说明:[`docs/reports/GO_STATUS.md`](docs/reports/GO_STATUS.md)(G-008 已关闭:同步 CLI = Rust `bat`)。
|
- Go 边界权威说明:[`docs/reports/GO_STATUS.md`](docs/reports/GO_STATUS.md)(G-008 已关闭:同步 CLI = Rust `bat`)。
|
||||||
- 官方同步会维护 `<output>/official-version-state.json`,明确记录当前已完成版本、正在拉取版本、上一个可用版本和失败版本。
|
- 官方同步会维护 `<output>/official-version-state.json`,明确记录当前已完成版本、正在拉取版本、上一个可用版本和失败版本。
|
||||||
- 资源导入链路可配置为在官方 release 发布后写入 CAS + `ResourceRepository`,资源 metadata 会记录 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式,TextAsset/Table/Media 会按类型分类索引;`resource.index` RPC/CLI 可按类型、hash、路径模式、官方 release ID、平台、destination、archive entry、parse status 和 TextUnit format 分页查询索引。
|
- 资源导入链路可配置为在官方 release 发布后写入 CAS + `ResourceRepository`,资源 metadata 会记录 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式,TextAsset/Table/Media 会按类型分类索引;`resource.index` RPC/CLI 可按类型、hash、路径模式、官方 release ID、平台、destination、archive entry、parse status 和 TextUnit format 分页查询索引。
|
||||||
- 新 release 发布后会生成 `official-resource-changes.json`、`official-parse-cache.json`、`official-textunit-index.json`、`official-textunit-tasks.json`、`crowdin-translation-handoff.json` 和 `crowdin-textunit-queue.json`;其中 TextUnit/Crowdin 队列只使用 Added/Modified 资源,不调用 Crowdin 网络 API,离线 TextUnit 翻译任务可通过 `translation.tasks` RPC/CLI 查询状态和跳过/失败原因。
|
- 新 release 发布后会生成 `official-resource-changes.json`、`official-parse-cache.json`、`official-textunit-index.json`、`official-textunit-tasks.json`、`crowdin-translation-handoff.json`、`crowdin-textunit-queue.json`、`translation-tasks.sqlite` 和 `translation-handoff.json`;其中 TextUnit/Crowdin 队列只使用 Added/Modified 资源,不调用 Crowdin 网络 API,离线 TextUnit 翻译任务可通过 `translation.tasks` / `translation.handoff` RPC 或 CLI 查询状态、跳过/失败原因和 provider run 交接。
|
||||||
- `LocalizedPatchService` 已具备 UnityFS TextAsset patch 发布前置能力:在 `--localized-output` / `BAT_LOCALIZED_OUTPUT` 配置的独立汉化目录 staging 中复制官方 release、应用 TextAsset patch、写 `localized-patch-manifest.json`(hash、size、diff、rollback)、校验后发布到 `versions/<id>` 并切换 `current`。
|
- `LocalizedPatchService` 已具备 UnityFS TextAsset patch 发布前置能力:在 `--localized-output` / `BAT_LOCALIZED_OUTPUT` 配置的独立汉化目录 staging 中复制官方 release、应用 TextAsset patch、写 `localized-patch-manifest.json`(hash、size、diff、rollback)、校验后发布到 `versions/<id>` 并切换 `current`。
|
||||||
- `bat-patch` 已具备通用 Patch 基础:确定性 Binary hunk diff/apply、RFC 6902 JSON Patch apply、UTF-8 Text Patch、Patch manifest、BLAKE3/size 完整性校验和 rollback 元数据;文件级 `patch.apply` RPC / `patch-apply` CLI 与 UnityFS TextAsset / TypeTree string / TypeTree 语义字段写入入口已开放,TypeTree 语义字段支持基础标量、固定 Unity float/int/hash 值类型的 leaf/direct-child 形态、PPtr、managed-reference registry payload 字符串、object 字段组合、unknown fixed-size raw bytes 同长度替换和 TypeTree schema 支撑的 array/vector/map 整体替换;TextUnit 提取会把 managed-reference 类型信息保留为上下文而非翻译文本,汉化发布当前仍走 UnityFS TextAsset 前置链路。
|
- `bat-patch` 已具备通用 Patch 基础:确定性 Binary hunk diff/apply、RFC 6902 JSON Patch apply、UTF-8 Text Patch、Patch manifest、BLAKE3/size 完整性校验和 rollback 元数据;文件级 `patch.apply` RPC / `patch-apply` CLI 与 UnityFS TextAsset / TypeTree string / TypeTree 语义字段写入入口已开放,TypeTree 语义字段支持基础标量、固定 Unity float/int/hash 值类型的 leaf/direct-child 形态、PPtr、managed-reference registry payload 字符串、object 字段组合、unknown fixed-size raw bytes 同长度替换和 TypeTree schema 支撑的 array/vector/map 整体替换;TextUnit 提取会把 managed-reference 类型信息保留为上下文而非翻译文本,汉化发布当前仍走 UnityFS TextAsset 前置链路。
|
||||||
- `bat-ffi` 可选无状态 C ABI 兼容层:仅保留 Manifest inspect 和官方 sync plan 的粗粒度 JSON helper,不作为 Go CLI 或生产同步的主集成边界。
|
- `bat-ffi` 可选无状态 C ABI 兼容层:仅保留 Manifest inspect 和官方 sync plan 的粗粒度 JSON helper,不作为 Go CLI 或生产同步的主集成边界。
|
||||||
|
|||||||
@@ -2,6 +2,45 @@
|
|||||||
|
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
use bat_core::domain::{GameClient, GameRegion};
|
use bat_core::domain::{GameClient, GameRegion};
|
||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
/// Linux-first client discovery backed by explicitly supplied roots.
|
||||||
|
///
|
||||||
|
/// The adapter never scans home directories implicitly and does not require
|
||||||
|
/// the official launcher. The roots are normally a staging/import directory
|
||||||
|
/// selected by the caller.
|
||||||
|
#[derive(Debug, Clone, Default)]
|
||||||
|
pub struct LinuxClientDiscovery {
|
||||||
|
roots: Vec<PathBuf>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LinuxClientDiscovery {
|
||||||
|
/// Creates a discovery adapter for explicit candidate roots.
|
||||||
|
pub fn new(roots: Vec<PathBuf>) -> Self {
|
||||||
|
Self { roots }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[async_trait]
|
||||||
|
impl ClientDiscovery for LinuxClientDiscovery {
|
||||||
|
async fn discover_all(&self) -> Result<Vec<GameClient>, String> {
|
||||||
|
GameClient::discover_in_roots(&self.roots).map_err(|error| error.to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn verify_client(&self, path: &str) -> bool {
|
||||||
|
GameClient::new(PathBuf::from(path), GameRegion::Japan)
|
||||||
|
.verify_integrity()
|
||||||
|
.unwrap_or(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn detect_region(&self, path: &str) -> Result<GameRegion, String> {
|
||||||
|
if self.verify_client(path).await {
|
||||||
|
Ok(GameRegion::Japan)
|
||||||
|
} else {
|
||||||
|
Err(format!("不是有效的 Linux Blue Archive 客户端:{path}"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// 客户端发现接口
|
/// 客户端发现接口
|
||||||
///
|
///
|
||||||
@@ -49,5 +88,44 @@ pub trait ClientDiscovery: Send + Sync {
|
|||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
// 测试将在实现时添加
|
use super::*;
|
||||||
|
use std::fs;
|
||||||
|
use tempfile::TempDir;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn linux_discovery_uses_explicit_roots_and_verifies_layout() {
|
||||||
|
let temp = TempDir::new().unwrap();
|
||||||
|
let client = temp.path().join("BlueArchive_JP");
|
||||||
|
fs::create_dir_all(client.join("BlueArchive_Data/StreamingAssets/AssetBundles")).unwrap();
|
||||||
|
let discovery = LinuxClientDiscovery::new(vec![temp.path().to_path_buf()]);
|
||||||
|
|
||||||
|
let clients = discovery.discover_all().await.unwrap();
|
||||||
|
assert_eq!(clients.len(), 1);
|
||||||
|
assert!(
|
||||||
|
discovery
|
||||||
|
.verify_client(clients[0].install_path.to_str().unwrap())
|
||||||
|
.await
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
discovery
|
||||||
|
.detect_region(clients[0].install_path.to_str().unwrap())
|
||||||
|
.await
|
||||||
|
.unwrap(),
|
||||||
|
GameRegion::Japan
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn linux_discovery_rejects_unrelated_path() {
|
||||||
|
let discovery = LinuxClientDiscovery::default();
|
||||||
|
assert!(
|
||||||
|
!discovery
|
||||||
|
.verify_client("/tmp/not-a-blue-archive-client")
|
||||||
|
.await
|
||||||
|
);
|
||||||
|
assert!(discovery
|
||||||
|
.detect_region("/tmp/not-a-blue-archive-client")
|
||||||
|
.await
|
||||||
|
.is_err());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,208 @@
|
|||||||
|
//! Official resource backend seams.
|
||||||
|
//!
|
||||||
|
//! The update pipeline consumes these small contracts instead of depending on
|
||||||
|
//! one region's URL and catalog rules everywhere. The JP implementation is
|
||||||
|
//! the only production adapter today; adding another region should implement
|
||||||
|
//! this module's contracts without changing downloader orchestration.
|
||||||
|
|
||||||
|
use super::inventory::{YostarJpPlatformCatalogInventory, YostarJpPlatformDownloadInventory};
|
||||||
|
use super::yostar_jp::{
|
||||||
|
is_official_yostar_jp_url, server_info_url, PatchPlatform, YostarJpResourceDiscoveryPlan,
|
||||||
|
YostarJpResourceRoot, YostarJpServerInfo,
|
||||||
|
};
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
/// Catalog bytes required to build one platform's download inventory.
|
||||||
|
#[derive(Debug, Clone, Copy)]
|
||||||
|
pub struct PlatformCatalogInput<'a> {
|
||||||
|
/// Platform represented by the catalog.
|
||||||
|
pub platform: PatchPlatform,
|
||||||
|
/// `BundlePackingInfo.bytes` payload.
|
||||||
|
pub bundle_packing_info: &'a [u8],
|
||||||
|
/// `MediaCatalog.bytes` payload.
|
||||||
|
pub media_catalog: &'a [u8],
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Region/backend contract used by official resource orchestration.
|
||||||
|
pub trait OfficialResourceBackend: Send + Sync {
|
||||||
|
/// Stable backend identifier persisted in diagnostics.
|
||||||
|
fn backend_id(&self) -> &'static str;
|
||||||
|
|
||||||
|
/// Builds the server-info URL from an official metadata file name.
|
||||||
|
fn server_info_url(&self, file_name: &str) -> Result<String, String>;
|
||||||
|
|
||||||
|
/// Selects a discovery plan from server-info and requested platforms.
|
||||||
|
fn discovery_plan(
|
||||||
|
&self,
|
||||||
|
server_info: &YostarJpServerInfo,
|
||||||
|
connection_group: &str,
|
||||||
|
app_version: &str,
|
||||||
|
platforms: &[PatchPlatform],
|
||||||
|
) -> Result<YostarJpResourceDiscoveryPlan, String>;
|
||||||
|
|
||||||
|
/// Builds platform-aware inventory from the verified seed catalog bytes.
|
||||||
|
fn inventory(
|
||||||
|
&self,
|
||||||
|
table_catalog: &[u8],
|
||||||
|
platform_catalogs: &[PlatformCatalogInput<'_>],
|
||||||
|
) -> YostarJpPlatformDownloadInventory;
|
||||||
|
|
||||||
|
/// Validates that a URL belongs to this backend's official hosts.
|
||||||
|
fn is_official_url(&self, url: &str) -> bool;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// URL-to-destination mapping contract for a resource backend.
|
||||||
|
pub trait DownloadUrlMapper: Send + Sync {
|
||||||
|
/// Maps an official HTTPS URL to a relative release destination.
|
||||||
|
fn relative_destination(&self, url: &str) -> Result<PathBuf, String>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The currently supported official Blue Archive JP backend.
|
||||||
|
#[derive(Debug, Clone, Copy, Default)]
|
||||||
|
pub struct YostarJpBackend;
|
||||||
|
|
||||||
|
impl OfficialResourceBackend for YostarJpBackend {
|
||||||
|
fn backend_id(&self) -> &'static str {
|
||||||
|
"bluearchive.yostar.jp"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn server_info_url(&self, file_name: &str) -> Result<String, String> {
|
||||||
|
server_info_url(file_name)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn discovery_plan(
|
||||||
|
&self,
|
||||||
|
server_info: &YostarJpServerInfo,
|
||||||
|
connection_group: &str,
|
||||||
|
app_version: &str,
|
||||||
|
platforms: &[PatchPlatform],
|
||||||
|
) -> Result<YostarJpResourceDiscoveryPlan, String> {
|
||||||
|
server_info.discovery_plan(connection_group, app_version, platforms)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn inventory(
|
||||||
|
&self,
|
||||||
|
table_catalog: &[u8],
|
||||||
|
platform_catalogs: &[PlatformCatalogInput<'_>],
|
||||||
|
) -> YostarJpPlatformDownloadInventory {
|
||||||
|
let catalogs = platform_catalogs
|
||||||
|
.iter()
|
||||||
|
.map(|catalog| {
|
||||||
|
YostarJpPlatformCatalogInventory::from_catalog_bytes(
|
||||||
|
catalog.platform,
|
||||||
|
catalog.bundle_packing_info,
|
||||||
|
catalog.media_catalog,
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
YostarJpPlatformDownloadInventory::from_catalog_bytes(table_catalog, catalogs)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_official_url(&self, url: &str) -> bool {
|
||||||
|
is_official_yostar_jp_url(url)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DownloadUrlMapper for YostarJpBackend {
|
||||||
|
fn relative_destination(&self, url: &str) -> Result<PathBuf, String> {
|
||||||
|
let rest = url
|
||||||
|
.strip_prefix("https://")
|
||||||
|
.ok_or_else(|| format!("官方 URL 必须使用 https:{url}"))?;
|
||||||
|
let (host, path) = rest
|
||||||
|
.split_once('/')
|
||||||
|
.ok_or_else(|| format!("官方 URL 缺少路径:{url}"))?;
|
||||||
|
let mut destination = PathBuf::from(sanitize_component(host, url)?);
|
||||||
|
for segment in path.split('/') {
|
||||||
|
if segment.is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
destination.push(sanitize_component(segment, url)?);
|
||||||
|
}
|
||||||
|
Ok(destination)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl YostarJpBackend {
|
||||||
|
/// Returns the validated resource-root builder for an official root.
|
||||||
|
pub fn resource_root(&self, addressables_root: &str) -> Result<YostarJpResourceRoot, String> {
|
||||||
|
YostarJpResourceRoot::from_addressables_root(addressables_root)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn sanitize_component(component: &str, url: &str) -> Result<String, String> {
|
||||||
|
if component == "." || component == ".." || component.is_empty() {
|
||||||
|
return Err(format!("官方 URL 包含不安全路径片段:{url}"));
|
||||||
|
}
|
||||||
|
if component.contains('?') || component.contains('#') || component.contains('\\') {
|
||||||
|
return Err(format!("官方资源 URL 包含不安全路径字符:{url}"));
|
||||||
|
}
|
||||||
|
Ok(component.to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Joins a backend-relative destination below an output root.
|
||||||
|
pub fn destination_under_root(root: &Path, relative: &Path) -> Result<PathBuf, String> {
|
||||||
|
if relative.is_absolute() {
|
||||||
|
return Err(format!(
|
||||||
|
"backend destination must be relative: {}",
|
||||||
|
relative.display()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let destination = root.join(relative);
|
||||||
|
if destination
|
||||||
|
.components()
|
||||||
|
.any(|component| matches!(component, std::path::Component::ParentDir))
|
||||||
|
{
|
||||||
|
return Err(format!(
|
||||||
|
"backend destination escapes output root: {}",
|
||||||
|
relative.display()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
Ok(destination)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn jp_backend_keeps_discovery_and_inventory_rules_in_one_adapter() {
|
||||||
|
let backend = YostarJpBackend;
|
||||||
|
let server_info = YostarJpServerInfo::from_json(
|
||||||
|
r#"{"ConnectionGroups":[{"Name":"Prod","AddressablesCatalogUrlRoot":"https://prod-clientpatch.bluearchiveyostar.com/r93_fixture"}]}"#,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let plan = backend
|
||||||
|
.discovery_plan(&server_info, "Prod", "1.70.0", &[PatchPlatform::Windows])
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(backend.backend_id(), "bluearchive.yostar.jp");
|
||||||
|
assert!(backend.is_official_url(&plan.endpoints[0].url));
|
||||||
|
|
||||||
|
let inventory = backend.inventory(
|
||||||
|
b"ExcelDB.db ExcelDB.db",
|
||||||
|
&[PlatformCatalogInput {
|
||||||
|
platform: PatchPlatform::Windows,
|
||||||
|
bundle_packing_info: b"FullPatch_000.zip",
|
||||||
|
media_catalog: b"GameData/Audio/JP.zip",
|
||||||
|
}],
|
||||||
|
);
|
||||||
|
assert_eq!(inventory.table_file_names, vec!["ExcelDB.db"]);
|
||||||
|
assert_eq!(inventory.platform_catalogs.len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn jp_backend_maps_and_rejects_unsafe_destinations() {
|
||||||
|
let backend = YostarJpBackend;
|
||||||
|
assert_eq!(
|
||||||
|
backend
|
||||||
|
.relative_destination(
|
||||||
|
"https://prod-clientpatch.bluearchiveyostar.com/r93/TableBundles/a.bytes"
|
||||||
|
)
|
||||||
|
.unwrap(),
|
||||||
|
PathBuf::from("prod-clientpatch.bluearchiveyostar.com/r93/TableBundles/a.bytes")
|
||||||
|
);
|
||||||
|
assert!(backend
|
||||||
|
.relative_destination("https://prod-clientpatch.bluearchiveyostar.com/r93/../secret")
|
||||||
|
.is_err());
|
||||||
|
assert!(!backend.is_official_url("https://example.invalid/a"));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -4,11 +4,16 @@
|
|||||||
//! client endpoints. Mirror-specific layers such as `bluearchive.cafe` or
|
//! client endpoints. Mirror-specific layers such as `bluearchive.cafe` or
|
||||||
//! `text=jp/voice=jp/media=jp` are intentionally excluded.
|
//! `text=jp/voice=jp/media=jp` are intentionally excluded.
|
||||||
|
|
||||||
|
pub mod backend;
|
||||||
pub mod game_main_config;
|
pub mod game_main_config;
|
||||||
pub mod inventory;
|
pub mod inventory;
|
||||||
pub mod launcher;
|
pub mod launcher;
|
||||||
pub mod yostar_jp;
|
pub mod yostar_jp;
|
||||||
|
|
||||||
|
pub use backend::{
|
||||||
|
destination_under_root, DownloadUrlMapper, OfficialResourceBackend, PlatformCatalogInput,
|
||||||
|
YostarJpBackend,
|
||||||
|
};
|
||||||
pub use game_main_config::YostarJpGameMainConfig;
|
pub use game_main_config::YostarJpGameMainConfig;
|
||||||
pub use inventory::{
|
pub use inventory::{
|
||||||
YostarJpDownloadInventory, YostarJpPlatformCatalogInventory, YostarJpPlatformDownloadInventory,
|
YostarJpDownloadInventory, YostarJpPlatformCatalogInventory, YostarJpPlatformDownloadInventory,
|
||||||
|
|||||||
+130
-19
@@ -1,6 +1,11 @@
|
|||||||
//! 游戏客户端领域对象
|
//! 游戏客户端领域对象
|
||||||
|
|
||||||
use std::path::PathBuf;
|
use std::collections::HashSet;
|
||||||
|
use std::env;
|
||||||
|
use std::fs;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
const CLIENT_ROOTS_ENV: &str = "BAT_CLIENT_ROOTS";
|
||||||
|
|
||||||
/// 游戏区域
|
/// 游戏区域
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)]
|
||||||
@@ -69,18 +74,58 @@ impl GameClient {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// 发现本地安装的客户端
|
/// 发现显式配置根目录下的本地客户端。
|
||||||
///
|
///
|
||||||
/// # 返回
|
/// # 返回
|
||||||
/// - 成功:返回找到的所有客户端
|
/// - 成功:返回找到的所有客户端
|
||||||
/// - 失败:返回错误
|
/// - 失败:返回错误
|
||||||
///
|
///
|
||||||
/// # 注意
|
/// 默认不扫描系统目录。调用方必须通过 `BAT_CLIENT_ROOTS` 提供一个或
|
||||||
/// 此功能将在 Phase 3 实现
|
/// 多个路径;路径格式使用平台原生路径分隔符。没有配置时返回空列表。
|
||||||
pub fn discover() -> crate::Result<Vec<GameClient>> {
|
pub fn discover() -> crate::Result<Vec<GameClient>> {
|
||||||
Err(crate::Error::NotImplemented(
|
let Some(value) = env::var_os(CLIENT_ROOTS_ENV) else {
|
||||||
"客户端发现功能将在 Phase 3 实现".to_string(),
|
return Ok(Vec::new());
|
||||||
))
|
};
|
||||||
|
let roots = env::split_paths(&value).collect::<Vec<_>>();
|
||||||
|
Self::discover_in_roots(&roots)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// 在调用方明确提供的隔离根目录下发现客户端。
|
||||||
|
///
|
||||||
|
/// 每个根目录只检查根本身和它的直接子目录,不递归扫描用户目录。
|
||||||
|
/// 当前核心模型的默认发现区域为日本服;其他区域应由适配器提供
|
||||||
|
/// 专用区域识别策略。
|
||||||
|
pub fn discover_in_roots(roots: &[PathBuf]) -> crate::Result<Vec<GameClient>> {
|
||||||
|
let mut candidates = Vec::new();
|
||||||
|
let mut seen = HashSet::new();
|
||||||
|
|
||||||
|
for root in roots {
|
||||||
|
if !is_real_directory(root)? || has_symlink_component(root)? {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if seen.insert(root.clone()) {
|
||||||
|
candidates.push(root.clone());
|
||||||
|
}
|
||||||
|
|
||||||
|
for entry in fs::read_dir(root)? {
|
||||||
|
let entry = entry?;
|
||||||
|
let path = entry.path();
|
||||||
|
if !is_real_directory(&path)? || has_symlink_component(&path)? {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if seen.insert(path.clone()) {
|
||||||
|
candidates.push(path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut clients = Vec::new();
|
||||||
|
for path in candidates {
|
||||||
|
if client_layout_is_present(&path)? {
|
||||||
|
clients.push(GameClient::new(path, GameRegion::Japan));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(clients)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// 验证客户端完整性
|
/// 验证客户端完整性
|
||||||
@@ -89,12 +134,11 @@ impl GameClient {
|
|||||||
/// - true: 客户端完整
|
/// - true: 客户端完整
|
||||||
/// - false: 客户端损坏
|
/// - false: 客户端损坏
|
||||||
///
|
///
|
||||||
/// # 注意
|
|
||||||
/// 此功能将在 Phase 3 实现
|
|
||||||
pub fn verify_integrity(&self) -> crate::Result<bool> {
|
pub fn verify_integrity(&self) -> crate::Result<bool> {
|
||||||
Err(crate::Error::NotImplemented(
|
if !is_real_directory(&self.install_path)? || has_symlink_component(&self.install_path)? {
|
||||||
"完整性验证将在 Phase 3 实现".to_string(),
|
return Ok(false);
|
||||||
))
|
}
|
||||||
|
Ok(client_layout_is_present(&self.install_path)?)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// 获取 StreamingAssets 目录路径
|
/// 获取 StreamingAssets 目录路径
|
||||||
@@ -113,6 +157,8 @@ impl GameClient {
|
|||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
use std::fs;
|
||||||
|
use tempfile::TempDir;
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_game_region_code() {
|
fn test_game_region_code() {
|
||||||
@@ -153,12 +199,77 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_discover_not_implemented() {
|
fn test_discover_without_explicit_roots_is_empty() {
|
||||||
let result = GameClient::discover();
|
// discover() 不得因为测试机或用户 home 中存在目录而扫描它们。
|
||||||
assert!(result.is_err());
|
assert!(GameClient::discover_in_roots(&[]).unwrap().is_empty());
|
||||||
assert!(matches!(
|
}
|
||||||
result.unwrap_err(),
|
|
||||||
crate::Error::NotImplemented(_)
|
#[test]
|
||||||
));
|
fn test_discover_and_verify_isolated_client_layout() {
|
||||||
|
let temp = TempDir::new().unwrap();
|
||||||
|
let client_root = temp.path().join("BlueArchive_JP");
|
||||||
|
fs::create_dir_all(client_root.join("BlueArchive_Data/StreamingAssets/AssetBundles"))
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let clients = GameClient::discover_in_roots(&[temp.path().to_path_buf()]).unwrap();
|
||||||
|
assert_eq!(clients.len(), 1);
|
||||||
|
assert_eq!(clients[0].install_path, client_root);
|
||||||
|
assert_eq!(clients[0].region, GameRegion::Japan);
|
||||||
|
assert!(clients[0].verify_integrity().unwrap());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_integrity_rejects_incomplete_layout() {
|
||||||
|
let temp = TempDir::new().unwrap();
|
||||||
|
let client = GameClient::new(temp.path().join("missing"), GameRegion::Japan);
|
||||||
|
assert!(!client.verify_integrity().unwrap());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(unix)]
|
||||||
|
#[test]
|
||||||
|
fn test_discovery_and_integrity_reject_symlinked_client() {
|
||||||
|
use std::os::unix::fs::symlink;
|
||||||
|
|
||||||
|
let temp = TempDir::new().unwrap();
|
||||||
|
let real = temp.path().join("real");
|
||||||
|
fs::create_dir_all(real.join("BlueArchive_Data/StreamingAssets/AssetBundles")).unwrap();
|
||||||
|
let link = temp.path().join("link");
|
||||||
|
symlink(&real, &link).unwrap();
|
||||||
|
|
||||||
|
let clients = GameClient::discover_in_roots(&[temp.path().to_path_buf()]).unwrap();
|
||||||
|
assert_eq!(clients.len(), 1);
|
||||||
|
assert_eq!(clients[0].install_path, real);
|
||||||
|
assert!(!GameClient::new(link, GameRegion::Japan)
|
||||||
|
.verify_integrity()
|
||||||
|
.unwrap());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn client_layout_is_present(path: &Path) -> crate::Result<bool> {
|
||||||
|
Ok(is_real_directory(&path.join("BlueArchive_Data"))?
|
||||||
|
&& is_real_directory(&path.join("BlueArchive_Data/StreamingAssets"))?
|
||||||
|
&& is_real_directory(&path.join("BlueArchive_Data/StreamingAssets/AssetBundles"))?
|
||||||
|
&& !has_symlink_component(path)?)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_real_directory(path: &Path) -> crate::Result<bool> {
|
||||||
|
match fs::symlink_metadata(path) {
|
||||||
|
Ok(metadata) => Ok(metadata.is_dir() && !metadata.file_type().is_symlink()),
|
||||||
|
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(false),
|
||||||
|
Err(error) => Err(error.into()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn has_symlink_component(path: &Path) -> crate::Result<bool> {
|
||||||
|
let mut current = PathBuf::new();
|
||||||
|
for component in path.components() {
|
||||||
|
current.push(component.as_os_str());
|
||||||
|
match fs::symlink_metadata(¤t) {
|
||||||
|
Ok(metadata) if metadata.file_type().is_symlink() => return Ok(true),
|
||||||
|
Ok(_) => {}
|
||||||
|
Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(false),
|
||||||
|
Err(error) => return Err(error.into()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(false)
|
||||||
|
}
|
||||||
|
|||||||
@@ -146,6 +146,12 @@
|
|||||||
17. 旧 launcher 包或 `resources.assets` 下载使用官方 launcher CDN 配置,primary CDN 失败后切换 official backup CDN;资源 patch host 不猜测非官方镜像。
|
17. 旧 launcher 包或 `resources.assets` 下载使用官方 launcher CDN 配置,primary CDN 失败后切换 official backup CDN;资源 patch host 不猜测非官方镜像。
|
||||||
18. 记录最终文件大小、本次传输字节数、官方 hash 校验数和执行状态。
|
18. 记录最终文件大小、本次传输字节数、官方 hash 校验数和执行状态。
|
||||||
19. 非官方 URL 直接拒绝。
|
19. 非官方 URL 直接拒绝。
|
||||||
|
20. 下载调度默认并发数为 `8`,允许范围是 `1..=256`,由
|
||||||
|
`--download-concurrency` / `BAT_DOWNLOAD_CONCURRENCY` 配置。worker 从共享
|
||||||
|
plan 队列逐项领取任务,单个任务完成后立即领取下一个,不等待其他 worker
|
||||||
|
的当前任务;完成结果在协调线程即时更新 manifest、hash 事件和进度计数。
|
||||||
|
最终 `OfficialResourcePullReport.items` 仍按 `OfficialResourcePullPlan`
|
||||||
|
顺序排列,避免并发完成顺序泄露到发布和 API 读侧。
|
||||||
|
|
||||||
路径映射时会做分段清理,并在写入前做输出目录安全校验、相对路径归属校验和现有路径组件 symlink 检查,避免把不安全路径写进输出目录或通过 symlink 跳出输出目录。
|
路径映射时会做分段清理,并在写入前做输出目录安全校验、相对路径归属校验和现有路径组件 symlink 检查,避免把不安全路径写进输出目录或通过 symlink 跳出输出目录。
|
||||||
|
|
||||||
@@ -171,9 +177,12 @@
|
|||||||
|
|
||||||
这层的意义是把“下载到磁盘的文件”变成“可查询、可复用、可去重”的资源对象。
|
这层的意义是把“下载到磁盘的文件”变成“可查询、可复用、可去重”的资源对象。
|
||||||
`resource.index` RPC / CLI 只读查询现有 SQLite 索引;索引不存在时返回
|
`resource.index` RPC / CLI 只读查询现有 SQLite 索引;索引不存在时返回
|
||||||
`available=false`,不会因为查询创建空库。`translation.tasks` RPC / CLI
|
`available=false`,不会因为查询创建空库。发布后的 TextUnit 队列还会在当前
|
||||||
可查询离线 TextUnit 任务状态和跳过/失败 reason;G-011 剩余工作是真实
|
release 根目录写入 `translation-tasks.sqlite`,由版本化 `schema_migrations`
|
||||||
worker 状态写回、CAS 诊断入口和面向大索引的查询优化。
|
管理 queued/running/failed/completed/skipped 和 failure reason。`translation.tasks`
|
||||||
|
优先查询这份状态库,`translation.task.update` 供 provider worker 回写状态;
|
||||||
|
没有状态库的旧 release 才回退到 immutable JSON 队列。G-011 剩余工作是 CAS
|
||||||
|
诊断入口和面向大索引的查询优化。
|
||||||
|
|
||||||
对应实现主要在:
|
对应实现主要在:
|
||||||
|
|
||||||
@@ -226,7 +235,7 @@ worker 状态写回、CAS 诊断入口和面向大索引的查询优化。
|
|||||||
11. 下载、manifest、本地 BLAKE3、ZIP 和官方 `.hash` 校验完成后写入新的 snapshot,并在 staging 中写入 `official-launcher-bootstrap.json`(若本轮启用 `--auto-discover`)。
|
11. 下载、manifest、本地 BLAKE3、ZIP 和官方 `.hash` 校验完成后写入新的 snapshot,并在 staging 中写入 `official-launcher-bootstrap.json`(若本轮启用 `--auto-discover`)。
|
||||||
12. 将 staging rename 为 `<output>/versions/<id>`,再原子替换 `<output>/current` symlink 指向该 versioned 目录。
|
12. 将 staging rename 为 `<output>/versions/<id>`,再原子替换 `<output>/current` symlink 指向该 versioned 目录。
|
||||||
13. 发布完成后先对比上一完整 release 和当前 release 的 `official-download-manifest.json`,写出 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`。同一 destination 只有 size 或 BLAKE3 变化才算 modified;新增+变更资源进入解析/翻译 handoff,删除资源只进入差异记录。当前只预留 Crowdin 本地 handoff,不发外部 API 请求。
|
13. 发布完成后先对比上一完整 release 和当前 release 的 `official-download-manifest.json`,写出 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`。同一 destination 只有 size 或 BLAKE3 变化才算 modified;新增+变更资源进入解析/翻译 handoff,删除资源只进入差异记录。当前只预留 Crowdin 本地 handoff,不发外部 API 请求。
|
||||||
14. 随后刷新 active release 下的 `official-parse-cache.json` 和 `official-textunit-index.json`,并从 Added/Modified 资源、parse cache 与 TextUnit 明细索引派生 `official-textunit-tasks.json` 和 `crowdin-textunit-queue.json`;up-to-date 轮询在已有有效解析缓存、TextUnit 明细索引和 TextUnit 队列时只读取摘要,不重复解析。
|
14. 随后刷新 active release 下的 `official-parse-cache.json` 和 `official-textunit-index.json`,并从 Added/Modified 资源、parse cache 与 TextUnit 明细索引派生 `official-textunit-tasks.json`、`crowdin-textunit-queue.json` 和版本化的 `translation-tasks.sqlite`;up-to-date 轮询在已有有效解析缓存、TextUnit 明细索引和 TextUnit 队列时只读取摘要,不重复解析,重新同步队列时保留已有 worker 状态。
|
||||||
15. 若启用 `--import-repository`,已校验 release 会被导入 CAS + `ResourceRepository`,并可经 `resource.index` 查询。
|
15. 若启用 `--import-repository`,已校验 release 会被导入 CAS + `ResourceRepository`,并可经 `resource.index` 查询。
|
||||||
16. 官方同步报告默认给出 `localized_release_status=not_localized`,表示原版资源已发布、汉化资源未发布;UnityFS TextAsset patch 发布成功并通过 `localized-patch-manifest.json`、current symlink 和 release ID 校验后,`localized.status` 才返回 `localized`,表示原版和汉化两套资源都已发布。
|
16. 官方同步报告默认给出 `localized_release_status=not_localized`,表示原版资源已发布、汉化资源未发布;UnityFS TextAsset patch 发布成功并通过 `localized-patch-manifest.json`、current symlink 和 release ID 校验后,`localized.status` 才返回 `localized`,表示原版和汉化两套资源都已发布。
|
||||||
|
|
||||||
|
|||||||
@@ -219,12 +219,20 @@ cargo run -p bat-infrastructure --bin bat -- \
|
|||||||
`official-resource-changes.json`、`crowdin-translation-handoff.json`、
|
`official-resource-changes.json`、`crowdin-translation-handoff.json`、
|
||||||
`official-parse-cache.json` 和 `official-textunit-index.json`,随后从
|
`official-parse-cache.json` 和 `official-textunit-index.json`,随后从
|
||||||
Added/Modified 资源、parse cache 与 TextUnit 明细索引派生
|
Added/Modified 资源、parse cache 与 TextUnit 明细索引派生
|
||||||
`official-textunit-tasks.json` 和 `crowdin-textunit-queue.json`。本地已有旧完整
|
`official-textunit-tasks.json`、`crowdin-textunit-queue.json`、
|
||||||
|
`translation-tasks.sqlite` 和 `translation-handoff.json`。本地已有旧完整
|
||||||
版本时,新版本发布后会先按 manifest destination 对比旧/新 release,只把新增和
|
版本时,新版本发布后会先按 manifest destination 对比旧/新 release,只把新增和
|
||||||
内容变更的资源写入解析与 Crowdin handoff;删除资源只记录差异,不进入翻译队列。
|
内容变更的资源写入解析与 Crowdin handoff;删除资源只记录差异,不进入翻译队列。
|
||||||
up-to-date 轮询发现本地文件、解析缓存、TextUnit 明细索引和 TextUnit 队列未变时不会重复解析。
|
up-to-date 轮询发现本地文件、解析缓存、TextUnit 明细索引和 TextUnit 队列未变时不会重复解析。
|
||||||
Crowdin 队列当前只落本地文件,不发网络请求。
|
Crowdin 队列当前只落本地文件,不发网络请求。
|
||||||
|
|
||||||
|
官方下载服务默认使用 8 个有界 worker,`--download-concurrency` /
|
||||||
|
`BAT_DOWNLOAD_CONCURRENCY` 只接受 `1..=256`。worker 完成一个 URL 后立即从共享
|
||||||
|
队列领取下一个任务;finished 进度按实际完成顺序即时上报,完成计数单调递增,
|
||||||
|
最终 report 的资源列表仍按 pull plan 顺序。需要验证顺序模式时显式使用
|
||||||
|
`--download-concurrency 1`。本文档中的真实资源命令仅是隔离 runbook;本地轻量
|
||||||
|
验证应使用 fake-curl/fixture,不要在开发机执行真实下载或 smoke run。
|
||||||
|
|
||||||
需要把已校验官方 release 导入 CAS + `ResourceRepository` 时,显式启用:
|
需要把已校验官方 release 导入 CAS + `ResourceRepository` 时,显式启用:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -243,6 +251,8 @@ cargo run -p bat-infrastructure --bin bat -- parse-status
|
|||||||
cargo run -p bat-infrastructure --bin bat -- parse-text-units --limit 50
|
cargo run -p bat-infrastructure --bin bat -- parse-text-units --limit 50
|
||||||
cargo run -p bat-infrastructure --bin bat -- parse-errors --limit 50
|
cargo run -p bat-infrastructure --bin bat -- parse-errors --limit 50
|
||||||
cargo run -p bat-infrastructure --bin bat -- translation-tasks --task-status skipped_parse_failed --has-reason --limit 50
|
cargo run -p bat-infrastructure --bin bat -- translation-tasks --task-status skipped_parse_failed --has-reason --limit 50
|
||||||
|
cargo run -p bat-infrastructure --bin bat -- translation-tasks --worker-status failed --has-failure-reason --limit 50
|
||||||
|
cargo run -p bat-infrastructure --bin bat -- translation-handoff
|
||||||
cargo run -p bat-infrastructure --bin bat -- localized-status
|
cargo run -p bat-infrastructure --bin bat -- localized-status
|
||||||
cargo run -p bat-infrastructure --bin bat -- resource-index --limit 50
|
cargo run -p bat-infrastructure --bin bat -- resource-index --limit 50
|
||||||
cargo run -p bat-infrastructure --bin bat -- resource-index --release-id <ID> --platform windows --archive-entry <PATH> --format json --limit 50
|
cargo run -p bat-infrastructure --bin bat -- resource-index --release-id <ID> --platform windows --archive-entry <PATH> --format json --limit 50
|
||||||
@@ -251,8 +261,13 @@ cargo run -p bat-infrastructure --bin bat -- resource-index --release-id <ID> --
|
|||||||
`parse-status` 会额外显示 TextUnit 明细索引和队列摘要;`parse-text-units` /
|
`parse-status` 会额外显示 TextUnit 明细索引和队列摘要;`parse-text-units` /
|
||||||
`parse-errors` 可按 destination、archive entry、path id、class id、field path
|
`parse-errors` 可按 destination、archive entry、path id、class id、field path
|
||||||
和 format 分页查询当前官方 release 的 TextUnit 明细与解析错误;
|
和 format 分页查询当前官方 release 的 TextUnit 明细与解析错误;
|
||||||
`translation-tasks` 可按 release、destination、archive entry、任务状态、parse status、
|
`translation-tasks` 可按 release、destination、archive entry、队列任务状态、provider
|
||||||
TextUnit format 和 reason presence 查询离线 TextUnit 翻译任务状态与跳过/失败原因;
|
worker 状态、parse status、TextUnit format、队列 reason 和 provider failure reason
|
||||||
|
查询离线 TextUnit 翻译任务状态与跳过/失败原因;发布后的状态保存在当前 release
|
||||||
|
根目录的 `translation-tasks.sqlite`,旧 release 没有状态库时回退到 JSON 队列;
|
||||||
|
`translation-handoff` / `translation.handoff` 会动态合并版本化
|
||||||
|
`translation-handoff.json` 与 SQLite 状态,返回 job、unit、provider run 的完整交接
|
||||||
|
视图;
|
||||||
`resource-index` 返回的资源 JSON 包含 release、平台、bundle path、TextAsset 和 TextUnit metadata,
|
`resource-index` 返回的资源 JSON 包含 release、平台、bundle path、TextAsset 和 TextUnit metadata,
|
||||||
并可按 release、平台、destination、bundle path、archive entry、parse status 和 TextUnit format 做资源级过滤;
|
并可按 release、平台、destination、bundle path、archive entry、parse status 和 TextUnit format 做资源级过滤;
|
||||||
`localized-status` 只有在 `localized-version-state.json`、`current` symlink 和
|
`localized-status` 只有在 `localized-version-state.json`、`current` symlink 和
|
||||||
|
|||||||
@@ -322,7 +322,7 @@ cargo run -p bat-infrastructure --bin bat -- \
|
|||||||
|
|
||||||
默认平台是 `Windows,Android`,无需显式传 `--platforms`;只有要覆盖默认平台时才传。`--interval` 是正常检查周期,默认 `1h`;watch/daemon 模式还会在每天北京时间(UTC+8)`03:00`、`16:00`、`18:00` 强制执行一次自动刷新,该轮会注入 `force=true`,并且会中断普通 interval 的 sleep。`--error-retry` 是下载、发现或校验失败后的重试周期,默认 `60s`,也可以用 `--error-retry-seconds 60`。CLI 默认启动时向 stderr 打印 `BlueArchiveToolkit` ASCII banner,并把阶段进度日志写到 stderr,包括自动发现、proxy、server-info、marker、catalog、audit、download、snapshot 和 publish 阶段;download 阶段会输出已完成计数和单文件开始/完成状态,下载执行保持顺序处理,已完成计数保持单调不倒退,audit 阶段会输出官方 `.hash`、本地 BLAKE3、需修复项和 ZIP 结构校验结果摘要。daemon 还会写 `bat-events.jsonl` 结构化日志并按大小轮转。命令结果默认以人类可读摘要写到 stdout。需要纯机器输出时加 `--json --no-progress`,需要显式开启进度日志则用 `--progress`;只想关闭横幅但保留日志时可加 `--no-banner`。错误时 stderr 输出 JSON error,watch 模式下错误 JSON 的 `next_retry_seconds` 使用失败重试周期;如果未关闭 progress,错误 JSON 前可能已有 banner 和进度日志。普通错误 exit `1`,资源目录锁冲突 exit `75`,`verify` 或 `doctor` 发现问题也返回非 0。
|
默认平台是 `Windows,Android`,无需显式传 `--platforms`;只有要覆盖默认平台时才传。`--interval` 是正常检查周期,默认 `1h`;watch/daemon 模式还会在每天北京时间(UTC+8)`03:00`、`16:00`、`18:00` 强制执行一次自动刷新,该轮会注入 `force=true`,并且会中断普通 interval 的 sleep。`--error-retry` 是下载、发现或校验失败后的重试周期,默认 `60s`,也可以用 `--error-retry-seconds 60`。CLI 默认启动时向 stderr 打印 `BlueArchiveToolkit` ASCII banner,并把阶段进度日志写到 stderr,包括自动发现、proxy、server-info、marker、catalog、audit、download、snapshot 和 publish 阶段;download 阶段会输出已完成计数和单文件开始/完成状态,下载执行保持顺序处理,已完成计数保持单调不倒退,audit 阶段会输出官方 `.hash`、本地 BLAKE3、需修复项和 ZIP 结构校验结果摘要。daemon 还会写 `bat-events.jsonl` 结构化日志并按大小轮转。命令结果默认以人类可读摘要写到 stdout。需要纯机器输出时加 `--json --no-progress`,需要显式开启进度日志则用 `--progress`;只想关闭横幅但保留日志时可加 `--no-banner`。错误时 stderr 输出 JSON error,watch 模式下错误 JSON 的 `next_retry_seconds` 使用失败重试周期;如果未关闭 progress,错误 JSON 前可能已有 banner 和进度日志。普通错误 exit `1`,资源目录锁冲突 exit `75`,`verify` 或 `doctor` 发现问题也返回非 0。
|
||||||
|
|
||||||
生产可以直接运行 `--watch`,也可以用 `--daemon` 后台运行,或者用 systemd service、容器或 Go 进程守护它。cron/systemd timer 仍可调用单次模式,但不再是 Rust 自动更新的唯一方式。项目是否热更新、热重载或重启进程,由上层业务集成决定。生产官方资源目录应使用独立输出目录,不要指向现有客户端或人工维护的资源目录;上层读取原版资源时应读取 `--output/current`,不要读取 `.staging` 或 `versions` 中未切换的目录。汉化 Patch/导出应写入 `--localized-output`,并保留官方相对目录结构,不能写回 `--output/current`。发布状态分两档:`not_localized` 只发布原版资源、不发布汉化资源;`localized` 发布原版和汉化两套资源。非 dry-run 每轮会创建 `--output/.official-sync.lock`,防止并发写同一官方资源目录;live daemon 还会阻止前台写命令直接修改它正在管理的同一目录。
|
生产可以直接运行 `--watch`,也可以用 `--daemon` 后台运行,或者用 systemd service、容器或 Go 进程守护它。cron/systemd timer 仍可调用单次模式,但不再是 Rust 自动更新的唯一方式。下载默认并发 8,可用 `--download-concurrency` / `BAT_DOWNLOAD_CONCURRENCY` 配置为 `1..=256`;worker 动态领取共享 plan,finished 进度即时按完成数统计,发布 report 仍按 plan 顺序。项目是否热更新、热重载或重启进程,由上层业务集成决定。生产官方资源目录应使用独立输出目录,不要指向现有客户端或人工维护的资源目录;上层读取原版资源时应读取 `--output/current`,不要读取 `.staging` 或 `versions` 中未切换的目录。汉化 Patch/导出应写入 `--localized-output`,并保留官方相对目录结构,不能写回 `--output/current`。发布状态分两档:`not_localized` 只发布原版资源、不发布汉化资源;`localized` 发布原版和汉化两套资源。非 dry-run 每轮会创建 `--output/.official-sync.lock`,防止并发写同一官方资源目录;live daemon 还会阻止前台写命令直接修改它正在管理的同一目录。
|
||||||
|
|
||||||
需要只做探测时可以加 `--dry-run`。需要关闭本地 audit 或 repair 时可以显式使用 `--no-audit-local` 或 `--no-repair`,但生产同步默认应保持开启。
|
需要只做探测时可以加 `--dry-run`。需要关闭本地 audit 或 repair 时可以显式使用 `--no-audit-local` 或 `--no-repair`,但生产同步默认应保持开启。
|
||||||
|
|
||||||
|
|||||||
@@ -151,6 +151,11 @@ SQLite `ResourceRepository`,索引不存在时返回 `ok=true` 且
|
|||||||
TextUnit 明细索引派生,记录 TextUnit 任务、跳过原因和解析诊断。
|
TextUnit 明细索引派生,记录 TextUnit 任务、跳过原因和解析诊断。
|
||||||
- `crowdin-textunit-queue.json`:只包含已产生 TextUnit 的离线任务,预留给后续
|
- `crowdin-textunit-queue.json`:只包含已产生 TextUnit 的离线任务,预留给后续
|
||||||
Crowdin worker;当前不会发出网络请求。
|
Crowdin worker;当前不会发出网络请求。
|
||||||
|
- `translation-tasks.sqlite`:当前 release 的可变 worker 状态库,记录
|
||||||
|
queued / running / failed / completed / skipped、attempt count、provider run
|
||||||
|
ID 和 failure reason;schema 由 `schema_migrations` 版本表管理。
|
||||||
|
- `translation-handoff.json`:当前 release 的版本化 job/unit/provider run 交接
|
||||||
|
快照;worker 更新后的实时状态仍以 `translation-tasks.sqlite` 为准。
|
||||||
|
|
||||||
删除资源只进入 `official-resource-changes.json`,不进入 Crowdin handoff。
|
删除资源只进入 `official-resource-changes.json`,不进入 Crowdin handoff。
|
||||||
|
|
||||||
@@ -162,6 +167,8 @@ SQLite `ResourceRepository`,索引不存在时返回 `ok=true` 且
|
|||||||
| `parse.text_units` | 已实现 | `{ "offset": 0, "limit": 100, "destination": "*Table*", "archive_entry": "*.bytes", "path_id": 1, "class_id": 114, "field_path": "*Text*", "format": "json" }` | 当前官方 release 的 TextUnit 明细分页。 |
|
| `parse.text_units` | 已实现 | `{ "offset": 0, "limit": 100, "destination": "*Table*", "archive_entry": "*.bytes", "path_id": 1, "class_id": 114, "field_path": "*Text*", "format": "json" }` | 当前官方 release 的 TextUnit 明细分页。 |
|
||||||
| `parse.errors` | 已实现 | `{ "offset": 0, "limit": 100, "destination": "*Table*", "archive_entry": "*.bytes", "path_id": 1, "class_id": 114, "field_path": "*Text*", "format": "json" }` | 当前官方 release 的解析错误分页。 |
|
| `parse.errors` | 已实现 | `{ "offset": 0, "limit": 100, "destination": "*Table*", "archive_entry": "*.bytes", "path_id": 1, "class_id": 114, "field_path": "*Text*", "format": "json" }` | 当前官方 release 的解析错误分页。 |
|
||||||
| `translation.tasks` | 已实现 | `{ "offset": 0, "limit": 100, "task_id": "...", "release_id": "...", "destination": "...", "archive_entry": "...", "status": "skipped_parse_failed", "parse_status": "failed", "format": "json", "has_reason": true }` | 当前官方 release 的离线 TextUnit 翻译任务状态分页。 |
|
| `translation.tasks` | 已实现 | `{ "offset": 0, "limit": 100, "task_id": "...", "release_id": "...", "destination": "...", "archive_entry": "...", "status": "skipped_parse_failed", "parse_status": "failed", "format": "json", "has_reason": true }` | 当前官方 release 的离线 TextUnit 翻译任务状态分页。 |
|
||||||
|
| `translation.handoff` | 已实现 | `null` | 当前官方 release 的 job、unit、provider run 交接视图;动态合并队列和 SQLite worker 状态。 |
|
||||||
|
| `translation.task.update` | 已实现 | `{ "task_id": "...", "status": "failed", "failure_reason": "...", "provider_run_id": "..." }` | 写入当前 release 的 provider worker 状态,返回可回查任务记录。 |
|
||||||
|
|
||||||
`parse.status` 是只读查询;没有当前 release 或没有解析缓存时返回
|
`parse.status` 是只读查询;没有当前 release 或没有解析缓存时返回
|
||||||
`ok=true` 且 `data.available=false`。解析缓存来自官方原版资源目录,不读取
|
`ok=true` 且 `data.available=false`。解析缓存来自官方原版资源目录,不读取
|
||||||
@@ -195,15 +202,25 @@ offset 和 error。TypeTree-covered managed reference 字段会进入结构化
|
|||||||
完整 managed reference registry 等暂不支持结构会进入解析错误,而不是静默降级为
|
完整 managed reference registry 等暂不支持结构会进入解析错误,而不是静默降级为
|
||||||
低保真文本。
|
低保真文本。
|
||||||
|
|
||||||
`translation.tasks` 查询 `official-textunit-tasks.json`,用于查看离线
|
`translation.tasks` 优先查询当前 release 的 `translation-tasks.sqlite`,旧 release
|
||||||
TextUnit 翻译任务候选、跳过状态和失败/跳过原因;没有当前 release 或没有任务队列时
|
没有该文件时回退到 `official-textunit-tasks.json`;用于查看离线 TextUnit
|
||||||
返回 `ok=true` 且 `data.available=false`。过滤参数包括 `task_id`、
|
翻译任务候选和 provider worker 状态。没有当前 release 或没有任务队列时返回
|
||||||
|
`ok=true` 且 `data.available=false`。过滤参数包括 `task_id`、
|
||||||
`official_release_id`/`release_id`、`destination`、`path_pattern`、
|
`official_release_id`/`release_id`、`destination`、`path_pattern`、
|
||||||
`archive_entry`、`status`/`task_status`、`parse_status`、`text_unit_format`/`format`
|
`archive_entry`、`status`/`task_status`、`worker_status`、`parse_status`、
|
||||||
和 `has_reason`。`entries[]` 会包含 `official_release_id`、`destination`、
|
`text_unit_format`/`format`、`has_reason` 和 `has_failure_reason`。
|
||||||
`archive_entry`、`parse_status`、`status`、`reason`、TextAsset/TextUnit 摘要和
|
`entries[]` 会包含 `official_release_id`、`destination`、`archive_entry`、
|
||||||
校验指纹。当前它只反映 Rust 侧离线 handoff 状态;真实 Crowdin worker 消费、
|
`parse_status`、队列 `status`、`task_status`、`failure_reason`、`attempt_count`、
|
||||||
重试和完成状态仍需后续 worker 模型写回后才能暴露。
|
`provider_run_id`、TextAsset/TextUnit 摘要和校验指纹。
|
||||||
|
|
||||||
|
`translation.task.update` 只更新当前 release 的 SQLite 状态库,不改写 immutable
|
||||||
|
队列文件,也不主动访问 Crowdin。`status` 支持 `queued`、`running`、`failed`、
|
||||||
|
`completed` 和 `skipped`;进入 `running` 会增加 attempt count,`completed` 会
|
||||||
|
记录完成时间,`failed` 可写入 `failure_reason`。因此 worker 消费 handoff 后,
|
||||||
|
bat-api 可通过 `translation.tasks` 查询单项任务,也可通过
|
||||||
|
`translation.handoff` 获取完整 job/unit/provider run 状态。`translation.handoff`
|
||||||
|
不会触发下载或 provider 网络请求;没有当前 release 或任务队列时返回
|
||||||
|
`data.available=false`。
|
||||||
|
|
||||||
### localized
|
### localized
|
||||||
|
|
||||||
|
|||||||
@@ -286,9 +286,12 @@
|
|||||||
- 官方 release 发布后会写出 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`,新增+变更资源进入解析/翻译 handoff,删除资源只进入差异记录。
|
- 官方 release 发布后会写出 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`,新增+变更资源进入解析/翻译 handoff,删除资源只进入差异记录。
|
||||||
- `Resource` metadata 已通过 SQLite `metadata_json` 兼容迁移保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式;`resource.index` 会返回这些 metadata。
|
- `Resource` metadata 已通过 SQLite `metadata_json` 兼容迁移保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式;`resource.index` 会返回这些 metadata。
|
||||||
- 官方 release 发布后会持久化 `official-textunit-index.json`,记录单条 TextUnit 和解析错误;`parse.text_units` / `parse.errors` RPC 和 `parse-text-units` / `parse-errors` CLI 可按 destination、archive entry、path id、class id、field path 和 format 分页过滤。
|
- 官方 release 发布后会持久化 `official-textunit-index.json`,记录单条 TextUnit 和解析错误;`parse.text_units` / `parse.errors` RPC 和 `parse-text-units` / `parse-errors` CLI 可按 destination、archive entry、path id、class id、field path 和 format 分页过滤。
|
||||||
- 官方 release 发布后会从 Added/Modified 资源、parse cache 和 TextUnit 明细索引派生 `official-textunit-tasks.json` 和 `crowdin-textunit-queue.json`;删除资源不会进入队列。
|
- 官方 release 发布后会从 Added/Modified 资源、parse cache 和 TextUnit 明细索引派生 `official-textunit-tasks.json`、`crowdin-textunit-queue.json` 和 `translation-tasks.sqlite`;删除资源不会进入队列。
|
||||||
- `translation.tasks` RPC/CLI 已能分页查询 `official-textunit-tasks.json` 中的离线 TextUnit 任务状态、parse 状态、TextUnit format 和跳过/失败 reason。
|
- 官方 release 发布后还会写入版本化 `translation-handoff.json`;`translation.handoff` RPC/CLI 动态合并该快照与 SQLite worker 状态,暴露 job、unit、provider run、attempt 和 failure reason。
|
||||||
- 真实 Crowdin worker 消费后的任务状态、重试次数、远端失败原因和完成结果查询仍需补齐。
|
- `translation-tasks.sqlite` 由 `schema_migrations` 管理 durable task state、attempt count、provider run ID 和 failure reason;重复同步会保留已有 worker 状态。
|
||||||
|
- `translation.tasks` RPC/CLI 优先查询 `translation-tasks.sqlite`,旧 release 没有状态库时回退到 `official-textunit-tasks.json`;返回队列 `status`、worker `task_status`、failure reason 和时间/尝试次数。
|
||||||
|
- `translation.task.update` RPC 已提供 queued/running/failed/completed/skipped 状态回写契约,provider worker 可在消费 handoff 后按 task_id 更新并由同一查询接口反查。
|
||||||
|
- 真实 Crowdin 网络 worker、翻译记忆和完整 localized repack 仍属于后续翻译系统工作,不在当前 Rust 离线状态仓储范围内。
|
||||||
|
|
||||||
验收:
|
验收:
|
||||||
|
|
||||||
@@ -554,7 +557,9 @@
|
|||||||
|
|
||||||
1. issue #24:失败 staging 复用回归已补;核对残余场景。
|
1. issue #24:失败 staging 复用回归已补;核对残余场景。
|
||||||
2. issue #1:RPC 主体、文件级 `patch.apply` / `unityfs.patch_text_asset` / `unityfs.patch_string_field` / `unityfs.patch_field` 已落地;剩余发布级 patch build/rollback、复杂 UnityFS 语义编辑与设计边界确认。
|
2. issue #1:RPC 主体、文件级 `patch.apply` / `unityfs.patch_text_asset` / `unityfs.patch_string_field` / `unityfs.patch_field` 已落地;剩余发布级 patch build/rollback、复杂 UnityFS 语义编辑与设计边界确认。
|
||||||
3. issue #17 及子 issue:已按 wontfix 关闭多线程下载(顺序下载 + 指数退避)。
|
3. issue #17 的历史顺序/重试契约仍保留;当前 issue #33/#35 已补有界 downloader
|
||||||
|
scheduler 和默认并发 8,范围 `1..=256`,worker 完成后立即领取下一个任务,
|
||||||
|
进度即时按完成数上报,最终 report 保持 plan 顺序。
|
||||||
4. **G-008:已决策关闭**(同步 CLI = Rust `bat`;见 `GO_STATUS.md`)。
|
4. **G-008:已决策关闭**(同步 CLI = Rust `bat`;见 `GO_STATUS.md`)。
|
||||||
5. **G-009 / issue #19**:资源 bootstrap/分发 MVP 已编码;优先服务器联调与索引实勘,非「从零实现」。
|
5. **G-009 / issue #19**:资源 bootstrap/分发 MVP 已编码;优先服务器联调与索引实勘,非「从零实现」。
|
||||||
6. issue #2 / G-007(P1):Addressables 可校验字段。
|
6. issue #2 / G-007(P1):Addressables 可校验字段。
|
||||||
|
|||||||
@@ -3,6 +3,8 @@ use bat_assetbundle::UnitySerializedReplacementValue;
|
|||||||
use bat_core::domain::{Resource, ResourceType};
|
use bat_core::domain::{Resource, ResourceType};
|
||||||
use bat_core::repositories::resource_repository::{ResourceQuery, ResourceRepository};
|
use bat_core::repositories::resource_repository::{ResourceQuery, ResourceRepository};
|
||||||
use bat_core::{ApiError, ErrorCode};
|
use bat_core::{ApiError, ErrorCode};
|
||||||
|
#[cfg(test)]
|
||||||
|
use bat_infrastructure::DEFAULT_DOWNLOAD_CONCURRENCY;
|
||||||
use bat_infrastructure::{
|
use bat_infrastructure::{
|
||||||
apply_patch_file, apply_unityfs_field_patch_file, apply_unityfs_string_field_patch_file,
|
apply_patch_file, apply_unityfs_field_patch_file, apply_unityfs_string_field_patch_file,
|
||||||
apply_unityfs_text_asset_patch_file, changed_endpoint_urls, diff_extended_snapshot,
|
apply_unityfs_text_asset_patch_file, changed_endpoint_urls, diff_extended_snapshot,
|
||||||
@@ -16,11 +18,12 @@ use bat_infrastructure::{
|
|||||||
OfficialUpdateConfig, OfficialUpdateProgress, OfficialUpdateReport, OfficialUpdateService,
|
OfficialUpdateConfig, OfficialUpdateProgress, OfficialUpdateReport, OfficialUpdateService,
|
||||||
OfficialUpdateSnapshot, OfficialUpdateStatus, OfficialVerificationSummary,
|
OfficialUpdateSnapshot, OfficialUpdateStatus, OfficialVerificationSummary,
|
||||||
OfficialVersionRecord, OfficialVersionState, PatchApplyKind, PatchApplyParams,
|
OfficialVersionRecord, OfficialVersionState, PatchApplyKind, PatchApplyParams,
|
||||||
PatchApplyReport, ReleaseFlowStatusCode, SqliteResourceRepository, UnityFsFieldPatchParams,
|
PatchApplyReport, ReleaseFlowStatusCode, SqliteResourceRepository,
|
||||||
|
SqliteTranslationTaskRepository, TranslationTaskStatus, UnityFsFieldPatchParams,
|
||||||
UnityFsPatchReport, UnityFsStringFieldPatchParams, UnityFsTextAssetPatchParams,
|
UnityFsPatchReport, UnityFsStringFieldPatchParams, UnityFsTextAssetPatchParams,
|
||||||
LOCALIZED_CURRENT_LINK, LOCALIZED_PATCH_MANIFEST_FILE, LOCALIZED_VERSIONS_DIR,
|
LOCALIZED_CURRENT_LINK, LOCALIZED_PATCH_MANIFEST_FILE, LOCALIZED_VERSIONS_DIR,
|
||||||
LOCALIZED_VERSION_STATE_FILE, OFFICIAL_PARSE_CACHE_FILE, OFFICIAL_TEXTUNIT_INDEX_FILE,
|
LOCALIZED_VERSION_STATE_FILE, MAX_DOWNLOAD_CONCURRENCY, MIN_DOWNLOAD_CONCURRENCY,
|
||||||
PRIVATE_FILE_MODE,
|
OFFICIAL_PARSE_CACHE_FILE, OFFICIAL_TEXTUNIT_INDEX_FILE, PRIVATE_FILE_MODE,
|
||||||
};
|
};
|
||||||
use serde::de::DeserializeOwned;
|
use serde::de::DeserializeOwned;
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
@@ -175,6 +178,7 @@ fn run() -> anyhow::Result<i32> {
|
|||||||
| CliCommand::ParseTextUnits
|
| CliCommand::ParseTextUnits
|
||||||
| CliCommand::ParseErrors
|
| CliCommand::ParseErrors
|
||||||
| CliCommand::TranslationTasks
|
| CliCommand::TranslationTasks
|
||||||
|
| CliCommand::TranslationHandoff
|
||||||
| CliCommand::LocalizedStatus
|
| CliCommand::LocalizedStatus
|
||||||
| CliCommand::ResourceIndex => {
|
| CliCommand::ResourceIndex => {
|
||||||
run_readonly_query_command(&options)?;
|
run_readonly_query_command(&options)?;
|
||||||
@@ -243,12 +247,14 @@ struct CliOptions {
|
|||||||
query_bundle_path: Option<String>,
|
query_bundle_path: Option<String>,
|
||||||
query_archive_entry: Option<String>,
|
query_archive_entry: Option<String>,
|
||||||
query_task_status: Option<String>,
|
query_task_status: Option<String>,
|
||||||
|
query_worker_status: Option<String>,
|
||||||
query_parse_status: Option<String>,
|
query_parse_status: Option<String>,
|
||||||
query_path_id: Option<i64>,
|
query_path_id: Option<i64>,
|
||||||
query_class_id: Option<i32>,
|
query_class_id: Option<i32>,
|
||||||
query_field_path: Option<String>,
|
query_field_path: Option<String>,
|
||||||
query_format: Option<String>,
|
query_format: Option<String>,
|
||||||
query_has_reason: Option<bool>,
|
query_has_reason: Option<bool>,
|
||||||
|
query_has_failure_reason: Option<bool>,
|
||||||
query_option_explicit: bool,
|
query_option_explicit: bool,
|
||||||
patch_kind: Option<PatchApplyKind>,
|
patch_kind: Option<PatchApplyKind>,
|
||||||
patch_source_path: Option<PathBuf>,
|
patch_source_path: Option<PathBuf>,
|
||||||
@@ -302,12 +308,14 @@ impl Default for CliOptions {
|
|||||||
query_bundle_path: None,
|
query_bundle_path: None,
|
||||||
query_archive_entry: None,
|
query_archive_entry: None,
|
||||||
query_task_status: None,
|
query_task_status: None,
|
||||||
|
query_worker_status: None,
|
||||||
query_parse_status: None,
|
query_parse_status: None,
|
||||||
query_path_id: None,
|
query_path_id: None,
|
||||||
query_class_id: None,
|
query_class_id: None,
|
||||||
query_field_path: None,
|
query_field_path: None,
|
||||||
query_format: None,
|
query_format: None,
|
||||||
query_has_reason: None,
|
query_has_reason: None,
|
||||||
|
query_has_failure_reason: None,
|
||||||
query_option_explicit: false,
|
query_option_explicit: false,
|
||||||
patch_kind: None,
|
patch_kind: None,
|
||||||
patch_source_path: None,
|
patch_source_path: None,
|
||||||
@@ -349,6 +357,7 @@ enum CliCommand {
|
|||||||
ParseTextUnits,
|
ParseTextUnits,
|
||||||
ParseErrors,
|
ParseErrors,
|
||||||
TranslationTasks,
|
TranslationTasks,
|
||||||
|
TranslationHandoff,
|
||||||
LocalizedStatus,
|
LocalizedStatus,
|
||||||
ResourceIndex,
|
ResourceIndex,
|
||||||
PatchApply,
|
PatchApply,
|
||||||
@@ -800,6 +809,8 @@ const RPC_METHOD_PARSE_STATUS: &str = "parse.status";
|
|||||||
const RPC_METHOD_PARSE_TEXT_UNITS: &str = "parse.text_units";
|
const RPC_METHOD_PARSE_TEXT_UNITS: &str = "parse.text_units";
|
||||||
const RPC_METHOD_PARSE_ERRORS: &str = "parse.errors";
|
const RPC_METHOD_PARSE_ERRORS: &str = "parse.errors";
|
||||||
const RPC_METHOD_TRANSLATION_TASKS: &str = "translation.tasks";
|
const RPC_METHOD_TRANSLATION_TASKS: &str = "translation.tasks";
|
||||||
|
const RPC_METHOD_TRANSLATION_HANDOFF: &str = "translation.handoff";
|
||||||
|
const RPC_METHOD_TRANSLATION_TASK_UPDATE: &str = "translation.task.update";
|
||||||
const RPC_METHOD_LOCALIZED_STATUS: &str = "localized.status";
|
const RPC_METHOD_LOCALIZED_STATUS: &str = "localized.status";
|
||||||
const RPC_METHOD_CATALOG_STATUS: &str = "catalog.status";
|
const RPC_METHOD_CATALOG_STATUS: &str = "catalog.status";
|
||||||
const RPC_METHOD_CATALOG_VERSIONS: &str = "catalog.versions";
|
const RPC_METHOD_CATALOG_VERSIONS: &str = "catalog.versions";
|
||||||
@@ -2278,6 +2289,16 @@ fn dispatch_rpc_method(
|
|||||||
build_translation_tasks_report(state_dir, query, offset, limit),
|
build_translation_tasks_report(state_dir, query, offset, limit),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
RPC_METHOD_TRANSLATION_HANDOFF => rpc_envelope_from_result(
|
||||||
|
request_id,
|
||||||
|
"translation.handoff",
|
||||||
|
build_translation_handoff_report(state_dir),
|
||||||
|
),
|
||||||
|
RPC_METHOD_TRANSLATION_TASK_UPDATE => rpc_envelope_from_result(
|
||||||
|
request_id,
|
||||||
|
"translation.task.update",
|
||||||
|
update_translation_task_status_report(state_dir, request.params.as_ref()),
|
||||||
|
),
|
||||||
RPC_METHOD_LOCALIZED_STATUS => rpc_envelope_from_result(
|
RPC_METHOD_LOCALIZED_STATUS => rpc_envelope_from_result(
|
||||||
request_id,
|
request_id,
|
||||||
"localized.status",
|
"localized.status",
|
||||||
@@ -2873,17 +2894,46 @@ fn query_resource_repository(
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn query_translation_task_repository(
|
||||||
|
repository_path: &Path,
|
||||||
|
query: &OfficialTextUnitTaskQuery,
|
||||||
|
offset: usize,
|
||||||
|
limit: usize,
|
||||||
|
) -> anyhow::Result<(u64, Vec<bat_infrastructure::PersistedTranslationTask>)> {
|
||||||
|
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||||
|
.enable_all()
|
||||||
|
.build()?;
|
||||||
|
runtime.block_on(async {
|
||||||
|
let repository = SqliteTranslationTaskRepository::open(repository_path)
|
||||||
|
.await
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))?;
|
||||||
|
let total_entries = repository
|
||||||
|
.count(query)
|
||||||
|
.await
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))?;
|
||||||
|
let entries = repository
|
||||||
|
.list(query)
|
||||||
|
.await
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))?
|
||||||
|
.into_iter()
|
||||||
|
.skip(offset)
|
||||||
|
.take(limit)
|
||||||
|
.collect();
|
||||||
|
Ok((total_entries, entries))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
fn repository_file_exists_no_symlink(path: &Path) -> anyhow::Result<bool> {
|
fn repository_file_exists_no_symlink(path: &Path) -> anyhow::Result<bool> {
|
||||||
|
sqlite_file_exists_no_symlink(path, "资源索引数据库")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn sqlite_file_exists_no_symlink(path: &Path, label: &str) -> anyhow::Result<bool> {
|
||||||
match fs::symlink_metadata(path) {
|
match fs::symlink_metadata(path) {
|
||||||
Ok(metadata) if metadata.file_type().is_symlink() => Err(anyhow::anyhow!(
|
Ok(metadata) if metadata.file_type().is_symlink() => {
|
||||||
"资源索引数据库不能是 symlink:{}",
|
Err(anyhow::anyhow!("{label}不能是 symlink:{}", path.display()))
|
||||||
path.display()
|
}
|
||||||
)),
|
|
||||||
Ok(metadata) if metadata.is_file() => Ok(true),
|
Ok(metadata) if metadata.is_file() => Ok(true),
|
||||||
Ok(_) => Err(anyhow::anyhow!(
|
Ok(_) => Err(anyhow::anyhow!("{label}不是普通文件:{}", path.display())),
|
||||||
"资源索引数据库不是普通文件:{}",
|
|
||||||
path.display()
|
|
||||||
)),
|
|
||||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(false),
|
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(false),
|
||||||
Err(error) => Err(error.into()),
|
Err(error) => Err(error.into()),
|
||||||
}
|
}
|
||||||
@@ -3089,6 +3139,8 @@ fn build_translation_tasks_report(
|
|||||||
let task_queue_path = record
|
let task_queue_path = record
|
||||||
.resource_root
|
.resource_root
|
||||||
.join(bat_infrastructure::OFFICIAL_TEXTUNIT_TASK_QUEUE_FILE);
|
.join(bat_infrastructure::OFFICIAL_TEXTUNIT_TASK_QUEUE_FILE);
|
||||||
|
let task_repository_path =
|
||||||
|
SqliteTranslationTaskRepository::repository_path(&record.resource_root);
|
||||||
let Some(queue) = bat_infrastructure::read_textunit_task_queue_at(&record.resource_root)
|
let Some(queue) = bat_infrastructure::read_textunit_task_queue_at(&record.resource_root)
|
||||||
.map_err(anyhow::Error::msg)?
|
.map_err(anyhow::Error::msg)?
|
||||||
else {
|
else {
|
||||||
@@ -3097,21 +3149,56 @@ fn build_translation_tasks_report(
|
|||||||
"current_version_id": record.id,
|
"current_version_id": record.id,
|
||||||
"resource_root": record.resource_root,
|
"resource_root": record.resource_root,
|
||||||
"textunit_task_queue_path": task_queue_path,
|
"textunit_task_queue_path": task_queue_path,
|
||||||
|
"task_repository_path": task_repository_path,
|
||||||
|
"task_repository_available": false,
|
||||||
}));
|
}));
|
||||||
};
|
};
|
||||||
let matches = bat_infrastructure::query_textunit_tasks(&queue, &query);
|
let task_repository_available =
|
||||||
let total_entries = matches.len();
|
sqlite_file_exists_no_symlink(&task_repository_path, "翻译任务状态数据库")?;
|
||||||
let entries = matches
|
let (total_entries, entries) = if task_repository_available {
|
||||||
.into_iter()
|
query_translation_task_repository(&task_repository_path, &query, offset, limit)?
|
||||||
.skip(offset)
|
} else {
|
||||||
.take(limit)
|
let mut queue_query = query.clone();
|
||||||
.cloned()
|
queue_query.task_status = None;
|
||||||
.collect::<Vec<_>>();
|
queue_query.has_failure_reason = None;
|
||||||
|
let matches = bat_infrastructure::query_textunit_tasks(&queue, &queue_query);
|
||||||
|
let persisted = matches
|
||||||
|
.into_iter()
|
||||||
|
.cloned()
|
||||||
|
.map(|task| {
|
||||||
|
bat_infrastructure::PersistedTranslationTask::from_queued_task(
|
||||||
|
task,
|
||||||
|
queue.generated_unix_seconds,
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.filter(|task| {
|
||||||
|
query
|
||||||
|
.task_status
|
||||||
|
.as_ref()
|
||||||
|
.is_none_or(|status| task.task_status.as_str() == status)
|
||||||
|
})
|
||||||
|
.filter(|task| {
|
||||||
|
query
|
||||||
|
.has_failure_reason
|
||||||
|
.is_none_or(|has_reason| task.failure_reason.is_some() == has_reason)
|
||||||
|
})
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
let total_entries = persisted.len();
|
||||||
|
let entries = persisted
|
||||||
|
.into_iter()
|
||||||
|
.skip(offset)
|
||||||
|
.take(limit)
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
(total_entries as u64, entries)
|
||||||
|
};
|
||||||
Ok(serde_json::json!({
|
Ok(serde_json::json!({
|
||||||
"available": true,
|
"available": true,
|
||||||
"current_version_id": record.id,
|
"current_version_id": record.id,
|
||||||
"resource_root": record.resource_root,
|
"resource_root": record.resource_root,
|
||||||
"textunit_task_queue_path": task_queue_path,
|
"textunit_task_queue_path": task_queue_path,
|
||||||
|
"task_repository_path": task_repository_path,
|
||||||
|
"task_repository_available": task_repository_available,
|
||||||
|
"task_repository_schema_version": bat_infrastructure::TRANSLATION_TASK_SCHEMA_VERSION,
|
||||||
"summary": queue.summary,
|
"summary": queue.summary,
|
||||||
"total_entries": total_entries,
|
"total_entries": total_entries,
|
||||||
"offset": offset,
|
"offset": offset,
|
||||||
@@ -3121,6 +3208,124 @@ fn build_translation_tasks_report(
|
|||||||
}))
|
}))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `translation.handoff`:当前已发布版本的动态翻译交接视图。
|
||||||
|
fn build_translation_handoff_report(state_dir: &Path) -> anyhow::Result<serde_json::Value> {
|
||||||
|
let (_, version_state) = read_daemon_resource_state(state_dir)?;
|
||||||
|
let current = version_state
|
||||||
|
.as_ref()
|
||||||
|
.and_then(|state| state.current_completed_version.as_ref());
|
||||||
|
let Some(record) = current else {
|
||||||
|
return Ok(serde_json::json!({ "available": false }));
|
||||||
|
};
|
||||||
|
let resource_root = &record.resource_root;
|
||||||
|
let task_queue_path = resource_root.join(bat_infrastructure::OFFICIAL_TEXTUNIT_TASK_QUEUE_FILE);
|
||||||
|
let handoff_path = resource_root.join(bat_infrastructure::TRANSLATION_HANDOFF_FILE);
|
||||||
|
let repository_path = SqliteTranslationTaskRepository::repository_path(resource_root);
|
||||||
|
let Some(queue) = bat_infrastructure::read_textunit_task_queue_at(resource_root)
|
||||||
|
.map_err(anyhow::Error::msg)?
|
||||||
|
else {
|
||||||
|
return Ok(serde_json::json!({
|
||||||
|
"available": false,
|
||||||
|
"current_version_id": record.id,
|
||||||
|
"resource_root": resource_root,
|
||||||
|
"textunit_task_queue_path": task_queue_path,
|
||||||
|
"translation_handoff_path": handoff_path,
|
||||||
|
"task_repository_path": repository_path,
|
||||||
|
}));
|
||||||
|
};
|
||||||
|
let task_repository_available =
|
||||||
|
sqlite_file_exists_no_symlink(&repository_path, "翻译任务状态数据库")?;
|
||||||
|
let tasks = if task_repository_available {
|
||||||
|
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||||
|
.enable_all()
|
||||||
|
.build()?;
|
||||||
|
runtime.block_on(async {
|
||||||
|
let repository = SqliteTranslationTaskRepository::open(&repository_path)
|
||||||
|
.await
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))?;
|
||||||
|
repository
|
||||||
|
.list(&OfficialTextUnitTaskQuery::default())
|
||||||
|
.await
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))
|
||||||
|
})?
|
||||||
|
} else {
|
||||||
|
queue
|
||||||
|
.tasks
|
||||||
|
.iter()
|
||||||
|
.cloned()
|
||||||
|
.map(|task| {
|
||||||
|
bat_infrastructure::PersistedTranslationTask::from_queued_task(
|
||||||
|
task,
|
||||||
|
queue.generated_unix_seconds,
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
};
|
||||||
|
let handoff = bat_infrastructure::build_translation_handoff(&queue, &tasks);
|
||||||
|
let handoff_file_available = sqlite_file_exists_no_symlink(&handoff_path, "翻译 handoff")?;
|
||||||
|
Ok(serde_json::json!({
|
||||||
|
"available": true,
|
||||||
|
"current_version_id": record.id,
|
||||||
|
"resource_root": resource_root,
|
||||||
|
"textunit_task_queue_path": task_queue_path,
|
||||||
|
"translation_handoff_path": handoff_path,
|
||||||
|
"translation_handoff_file_available": handoff_file_available,
|
||||||
|
"task_repository_path": repository_path,
|
||||||
|
"task_repository_available": task_repository_available,
|
||||||
|
"task_repository_schema_version": bat_infrastructure::TRANSLATION_TASK_SCHEMA_VERSION,
|
||||||
|
"handoff_schema_version": bat_infrastructure::TRANSLATION_HANDOFF_SCHEMA_VERSION,
|
||||||
|
"handoff": handoff,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `translation.task.update`:写入 provider worker 的可回查状态。
|
||||||
|
fn update_translation_task_status_report(
|
||||||
|
state_dir: &Path,
|
||||||
|
params: Option<&serde_json::Value>,
|
||||||
|
) -> anyhow::Result<serde_json::Value> {
|
||||||
|
let task_id = rpc_string_param(params, "task_id")
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("translation.task.update 缺少 task_id"))?;
|
||||||
|
let status_label = rpc_string_param(params, "status")
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("translation.task.update 缺少 status"))?;
|
||||||
|
let status = TranslationTaskStatus::parse(status_label)
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("不支持的翻译任务 worker 状态:{status_label}"))?;
|
||||||
|
let failure_reason = rpc_string_param(params, "failure_reason")
|
||||||
|
.or_else(|| rpc_string_param(params, "reason"))
|
||||||
|
.map(str::to_string);
|
||||||
|
let provider_run_id = rpc_string_param(params, "provider_run_id").map(str::to_string);
|
||||||
|
let (_, version_state) = read_daemon_resource_state(state_dir)?;
|
||||||
|
let current = version_state
|
||||||
|
.as_ref()
|
||||||
|
.and_then(|state| state.current_completed_version.as_ref())
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("没有可更新翻译任务的当前官方 release"))?;
|
||||||
|
let repository_path = SqliteTranslationTaskRepository::repository_path(¤t.resource_root);
|
||||||
|
if !sqlite_file_exists_no_symlink(&repository_path, "翻译任务状态数据库")? {
|
||||||
|
return Err(anyhow::anyhow!(
|
||||||
|
"翻译任务状态数据库不存在:{}",
|
||||||
|
repository_path.display()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||||
|
.enable_all()
|
||||||
|
.build()?;
|
||||||
|
let task = runtime.block_on(async {
|
||||||
|
let repository = SqliteTranslationTaskRepository::open(&repository_path)
|
||||||
|
.await
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))?;
|
||||||
|
repository
|
||||||
|
.update_status(task_id, status, failure_reason, provider_run_id)
|
||||||
|
.await
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))
|
||||||
|
})?;
|
||||||
|
Ok(serde_json::json!({
|
||||||
|
"available": true,
|
||||||
|
"current_version_id": current.id,
|
||||||
|
"task_repository_path": repository_path,
|
||||||
|
"entry": task,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
||||||
fn textunit_query_json(query: &OfficialTextUnitQuery) -> serde_json::Value {
|
fn textunit_query_json(query: &OfficialTextUnitQuery) -> serde_json::Value {
|
||||||
serde_json::json!({
|
serde_json::json!({
|
||||||
"destination": query.destination.clone(),
|
"destination": query.destination.clone(),
|
||||||
@@ -3141,9 +3346,11 @@ fn translation_task_query_json(query: &OfficialTextUnitTaskQuery) -> serde_json:
|
|||||||
"path_pattern": query.path_pattern.clone(),
|
"path_pattern": query.path_pattern.clone(),
|
||||||
"archive_entry": query.archive_entry.clone(),
|
"archive_entry": query.archive_entry.clone(),
|
||||||
"status": query.status.clone(),
|
"status": query.status.clone(),
|
||||||
|
"task_status": query.task_status.clone(),
|
||||||
"parse_status": query.parse_status.clone(),
|
"parse_status": query.parse_status.clone(),
|
||||||
"text_unit_format": query.text_unit_format.clone(),
|
"text_unit_format": query.text_unit_format.clone(),
|
||||||
"has_reason": query.has_reason,
|
"has_reason": query.has_reason,
|
||||||
|
"has_failure_reason": query.has_failure_reason,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -3398,11 +3605,13 @@ fn rpc_translation_task_query_params(
|
|||||||
status: rpc_string_param(params, "status")
|
status: rpc_string_param(params, "status")
|
||||||
.or_else(|| rpc_string_param(params, "task_status"))
|
.or_else(|| rpc_string_param(params, "task_status"))
|
||||||
.map(str::to_string),
|
.map(str::to_string),
|
||||||
|
task_status: rpc_string_param(params, "worker_status").map(str::to_string),
|
||||||
parse_status: rpc_string_param(params, "parse_status").map(str::to_string),
|
parse_status: rpc_string_param(params, "parse_status").map(str::to_string),
|
||||||
text_unit_format: rpc_string_param(params, "text_unit_format")
|
text_unit_format: rpc_string_param(params, "text_unit_format")
|
||||||
.or_else(|| rpc_string_param(params, "format"))
|
.or_else(|| rpc_string_param(params, "format"))
|
||||||
.map(str::to_string),
|
.map(str::to_string),
|
||||||
has_reason: rpc_bool_param(params, "has_reason"),
|
has_reason: rpc_bool_param(params, "has_reason"),
|
||||||
|
has_failure_reason: rpc_bool_param(params, "has_failure_reason"),
|
||||||
};
|
};
|
||||||
Ok((query, offset, limit))
|
Ok((query, offset, limit))
|
||||||
}
|
}
|
||||||
@@ -4357,6 +4566,7 @@ fn readonly_query_rpc_method(command: CliCommand) -> Option<&'static str> {
|
|||||||
CliCommand::ParseTextUnits => Some(RPC_METHOD_PARSE_TEXT_UNITS),
|
CliCommand::ParseTextUnits => Some(RPC_METHOD_PARSE_TEXT_UNITS),
|
||||||
CliCommand::ParseErrors => Some(RPC_METHOD_PARSE_ERRORS),
|
CliCommand::ParseErrors => Some(RPC_METHOD_PARSE_ERRORS),
|
||||||
CliCommand::TranslationTasks => Some(RPC_METHOD_TRANSLATION_TASKS),
|
CliCommand::TranslationTasks => Some(RPC_METHOD_TRANSLATION_TASKS),
|
||||||
|
CliCommand::TranslationHandoff => Some(RPC_METHOD_TRANSLATION_HANDOFF),
|
||||||
CliCommand::LocalizedStatus => Some(RPC_METHOD_LOCALIZED_STATUS),
|
CliCommand::LocalizedStatus => Some(RPC_METHOD_LOCALIZED_STATUS),
|
||||||
CliCommand::ResourceIndex => Some(RPC_METHOD_RESOURCE_INDEX),
|
CliCommand::ResourceIndex => Some(RPC_METHOD_RESOURCE_INDEX),
|
||||||
_ => None,
|
_ => None,
|
||||||
@@ -4471,6 +4681,9 @@ fn readonly_query_rpc_params(options: &CliOptions) -> Option<serde_json::Value>
|
|||||||
if let Some(status) = options.query_task_status.as_ref() {
|
if let Some(status) = options.query_task_status.as_ref() {
|
||||||
params.insert("status".to_string(), serde_json::json!(status));
|
params.insert("status".to_string(), serde_json::json!(status));
|
||||||
}
|
}
|
||||||
|
if let Some(status) = options.query_worker_status.as_ref() {
|
||||||
|
params.insert("worker_status".to_string(), serde_json::json!(status));
|
||||||
|
}
|
||||||
if let Some(parse_status) = options.query_parse_status.as_ref() {
|
if let Some(parse_status) = options.query_parse_status.as_ref() {
|
||||||
params.insert("parse_status".to_string(), serde_json::json!(parse_status));
|
params.insert("parse_status".to_string(), serde_json::json!(parse_status));
|
||||||
}
|
}
|
||||||
@@ -4480,6 +4693,12 @@ fn readonly_query_rpc_params(options: &CliOptions) -> Option<serde_json::Value>
|
|||||||
if let Some(has_reason) = options.query_has_reason {
|
if let Some(has_reason) = options.query_has_reason {
|
||||||
params.insert("has_reason".to_string(), serde_json::json!(has_reason));
|
params.insert("has_reason".to_string(), serde_json::json!(has_reason));
|
||||||
}
|
}
|
||||||
|
if let Some(has_failure_reason) = options.query_has_failure_reason {
|
||||||
|
params.insert(
|
||||||
|
"has_failure_reason".to_string(),
|
||||||
|
serde_json::json!(has_failure_reason),
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
_ => {}
|
_ => {}
|
||||||
}
|
}
|
||||||
@@ -4515,6 +4734,7 @@ fn build_readonly_query_report(
|
|||||||
options.query_offset,
|
options.query_offset,
|
||||||
options.query_limit,
|
options.query_limit,
|
||||||
),
|
),
|
||||||
|
RPC_METHOD_TRANSLATION_HANDOFF => build_translation_handoff_report(&options.state_dir),
|
||||||
RPC_METHOD_LOCALIZED_STATUS => {
|
RPC_METHOD_LOCALIZED_STATUS => {
|
||||||
build_localized_status_report(&options.state_dir, &options.config)
|
build_localized_status_report(&options.state_dir, &options.config)
|
||||||
}
|
}
|
||||||
@@ -4539,13 +4759,15 @@ fn validate_readonly_query_options(options: &CliOptions) -> anyhow::Result<()> {
|
|||||||
|| options.query_field_path.is_some();
|
|| options.query_field_path.is_some();
|
||||||
let has_translation_task_filter = options.query_task_id.is_some()
|
let has_translation_task_filter = options.query_task_id.is_some()
|
||||||
|| options.query_task_status.is_some()
|
|| options.query_task_status.is_some()
|
||||||
|| options.query_has_reason.is_some();
|
|| options.query_worker_status.is_some()
|
||||||
|
|| options.query_has_reason.is_some()
|
||||||
|
|| options.query_has_failure_reason.is_some();
|
||||||
|
|
||||||
match options.command {
|
match options.command {
|
||||||
CliCommand::ResourceIndex => {
|
CliCommand::ResourceIndex => {
|
||||||
if has_parse_object_filter || has_translation_task_filter {
|
if has_parse_object_filter || has_translation_task_filter {
|
||||||
return Err(anyhow::anyhow!(
|
return Err(anyhow::anyhow!(
|
||||||
"--path-id/--class-id/--field-path 只适用于 parse-text-units 或 parse-errors;--task-id/--task-status/--has-reason 只适用于 translation-tasks"
|
"--path-id/--class-id/--field-path 只适用于 parse-text-units 或 parse-errors;--task-id/--task-status/--worker-status/--has-reason/--has-failure-reason 只适用于 translation-tasks"
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -4556,7 +4778,7 @@ fn validate_readonly_query_options(options: &CliOptions) -> anyhow::Result<()> {
|
|||||||
|| options.query_parse_status.is_some()
|
|| options.query_parse_status.is_some()
|
||||||
{
|
{
|
||||||
return Err(anyhow::anyhow!(
|
return Err(anyhow::anyhow!(
|
||||||
"--resource-type/--hash/--release-id/--platform/--bundle-path/--parse-status 只适用于 resource-index 或 translation-tasks;--task-id/--task-status/--has-reason 只适用于 translation-tasks"
|
"--resource-type/--hash/--release-id/--platform/--bundle-path/--parse-status 只适用于 resource-index 或 translation-tasks;--task-id/--task-status/--worker-status/--has-reason/--has-failure-reason 只适用于 translation-tasks"
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -4567,6 +4789,11 @@ fn validate_readonly_query_options(options: &CliOptions) -> anyhow::Result<()> {
|
|||||||
));
|
));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
CliCommand::TranslationHandoff if options.query_option_explicit => {
|
||||||
|
return Err(anyhow::anyhow!(
|
||||||
|
"translation-handoff 不接受查询过滤参数;请使用 translation-tasks 查询单项任务"
|
||||||
|
));
|
||||||
|
}
|
||||||
CliCommand::ParseStatus | CliCommand::LocalizedStatus if options.query_option_explicit => {
|
CliCommand::ParseStatus | CliCommand::LocalizedStatus if options.query_option_explicit => {
|
||||||
return Err(anyhow::anyhow!(
|
return Err(anyhow::anyhow!(
|
||||||
"查询过滤参数只适用于 resource-index、parse-text-units、parse-errors 或 translation-tasks"
|
"查询过滤参数只适用于 resource-index、parse-text-units、parse-errors 或 translation-tasks"
|
||||||
@@ -4788,6 +5015,8 @@ fn translation_task_query_from_options(options: &CliOptions) -> OfficialTextUnit
|
|||||||
parse_status: options.query_parse_status.clone(),
|
parse_status: options.query_parse_status.clone(),
|
||||||
text_unit_format: options.query_format.clone(),
|
text_unit_format: options.query_format.clone(),
|
||||||
has_reason: options.query_has_reason,
|
has_reason: options.query_has_reason,
|
||||||
|
has_failure_reason: options.query_has_failure_reason,
|
||||||
|
task_status: options.query_worker_status.clone(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -6446,6 +6675,8 @@ fn daemon_child_args(options: &CliOptions) -> Vec<String> {
|
|||||||
}
|
}
|
||||||
args.push("--curl".to_string());
|
args.push("--curl".to_string());
|
||||||
args.push(config.curl_command.to_string_lossy().to_string());
|
args.push(config.curl_command.to_string_lossy().to_string());
|
||||||
|
args.push("--download-concurrency".to_string());
|
||||||
|
args.push(config.download_concurrency.to_string());
|
||||||
match config.curl_proxy.mode() {
|
match config.curl_proxy.mode() {
|
||||||
CurlProxyMode::Auto if options.proxy_option_explicit => {
|
CurlProxyMode::Auto if options.proxy_option_explicit => {
|
||||||
args.push("--proxy".to_string());
|
args.push("--proxy".to_string());
|
||||||
@@ -6960,6 +7191,8 @@ BAT_AUTO_DISCOVER=1
|
|||||||
# 逗号分隔:windows,android
|
# 逗号分隔:windows,android
|
||||||
#BAT_PLATFORMS=windows,android
|
#BAT_PLATFORMS=windows,android
|
||||||
#BAT_CURL=curl
|
#BAT_CURL=curl
|
||||||
|
# 最大并发下载数(默认 8;范围 1..=256)
|
||||||
|
#BAT_DOWNLOAD_CONCURRENCY=8
|
||||||
#BAT_UNZIP=unzip
|
#BAT_UNZIP=unzip
|
||||||
|
|
||||||
# ---- 输出 ----
|
# ---- 输出 ----
|
||||||
@@ -7138,6 +7371,10 @@ fn apply_bat_env_overrides(
|
|||||||
if let Some(v) = value("BAT_CURL") {
|
if let Some(v) = value("BAT_CURL") {
|
||||||
options.config.curl_command = PathBuf::from(v);
|
options.config.curl_command = PathBuf::from(v);
|
||||||
}
|
}
|
||||||
|
if let Some(v) = value("BAT_DOWNLOAD_CONCURRENCY") {
|
||||||
|
options.config.download_concurrency =
|
||||||
|
parse_download_concurrency(&v, "环境变量 BAT_DOWNLOAD_CONCURRENCY")?;
|
||||||
|
}
|
||||||
if let Some(v) = value("BAT_UNZIP") {
|
if let Some(v) = value("BAT_UNZIP") {
|
||||||
options.config.unzip_command = PathBuf::from(v);
|
options.config.unzip_command = PathBuf::from(v);
|
||||||
}
|
}
|
||||||
@@ -7245,6 +7482,10 @@ fn parse_args_with_env(
|
|||||||
ensure_command_not_set(options.command, "translation-tasks")?;
|
ensure_command_not_set(options.command, "translation-tasks")?;
|
||||||
options.command = CliCommand::TranslationTasks;
|
options.command = CliCommand::TranslationTasks;
|
||||||
}
|
}
|
||||||
|
"translation-handoff" => {
|
||||||
|
ensure_command_not_set(options.command, "translation-handoff")?;
|
||||||
|
options.command = CliCommand::TranslationHandoff;
|
||||||
|
}
|
||||||
"localized-status" => {
|
"localized-status" => {
|
||||||
ensure_command_not_set(options.command, "localized-status")?;
|
ensure_command_not_set(options.command, "localized-status")?;
|
||||||
options.command = CliCommand::LocalizedStatus;
|
options.command = CliCommand::LocalizedStatus;
|
||||||
@@ -7359,6 +7600,11 @@ fn parse_args_with_env(
|
|||||||
"--curl" => {
|
"--curl" => {
|
||||||
options.config.curl_command = PathBuf::from(next_option_value(&mut args, &flag)?);
|
options.config.curl_command = PathBuf::from(next_option_value(&mut args, &flag)?);
|
||||||
}
|
}
|
||||||
|
"--download-concurrency" => {
|
||||||
|
options.config.download_concurrency =
|
||||||
|
parse_download_concurrency(&next_option_value(&mut args, &flag)?, &flag)?;
|
||||||
|
options.sync_option_explicit = true;
|
||||||
|
}
|
||||||
"--proxy" => {
|
"--proxy" => {
|
||||||
options.config.curl_proxy =
|
options.config.curl_proxy =
|
||||||
parse_proxy_config(&next_option_value(&mut args, &flag)?)?;
|
parse_proxy_config(&next_option_value(&mut args, &flag)?)?;
|
||||||
@@ -7543,6 +7789,10 @@ fn parse_args_with_env(
|
|||||||
options.query_task_status = Some(next_option_value(&mut args, &flag)?);
|
options.query_task_status = Some(next_option_value(&mut args, &flag)?);
|
||||||
options.query_option_explicit = true;
|
options.query_option_explicit = true;
|
||||||
}
|
}
|
||||||
|
"--worker-status" => {
|
||||||
|
options.query_worker_status = Some(next_option_value(&mut args, &flag)?);
|
||||||
|
options.query_option_explicit = true;
|
||||||
|
}
|
||||||
"--parse-status" => {
|
"--parse-status" => {
|
||||||
options.query_parse_status = Some(next_option_value(&mut args, &flag)?);
|
options.query_parse_status = Some(next_option_value(&mut args, &flag)?);
|
||||||
options.query_option_explicit = true;
|
options.query_option_explicit = true;
|
||||||
@@ -7587,6 +7837,14 @@ fn parse_args_with_env(
|
|||||||
options.query_has_reason = Some(false);
|
options.query_has_reason = Some(false);
|
||||||
options.query_option_explicit = true;
|
options.query_option_explicit = true;
|
||||||
}
|
}
|
||||||
|
"--has-failure-reason" => {
|
||||||
|
options.query_has_failure_reason = Some(true);
|
||||||
|
options.query_option_explicit = true;
|
||||||
|
}
|
||||||
|
"--no-failure-reason" => {
|
||||||
|
options.query_has_failure_reason = Some(false);
|
||||||
|
options.query_option_explicit = true;
|
||||||
|
}
|
||||||
"--patch-kind" => {
|
"--patch-kind" => {
|
||||||
options.patch_kind = Some(parse_patch_apply_kind(&next_option_value(
|
options.patch_kind = Some(parse_patch_apply_kind(&next_option_value(
|
||||||
&mut args, &flag,
|
&mut args, &flag,
|
||||||
@@ -7704,6 +7962,7 @@ fn parse_args_with_env(
|
|||||||
| CliCommand::ParseTextUnits
|
| CliCommand::ParseTextUnits
|
||||||
| CliCommand::ParseErrors
|
| CliCommand::ParseErrors
|
||||||
| CliCommand::TranslationTasks
|
| CliCommand::TranslationTasks
|
||||||
|
| CliCommand::TranslationHandoff
|
||||||
| CliCommand::LocalizedStatus
|
| CliCommand::LocalizedStatus
|
||||||
| CliCommand::ResourceIndex => {
|
| CliCommand::ResourceIndex => {
|
||||||
validate_readonly_query_options(&options)?;
|
validate_readonly_query_options(&options)?;
|
||||||
@@ -7833,6 +8092,18 @@ fn tools_are_non_default(config: &OfficialUpdateConfig, baseline: &OfficialUpdat
|
|||||||
|| config.unzip_command != baseline.unzip_command
|
|| config.unzip_command != baseline.unzip_command
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn parse_download_concurrency(value: &str, source: &str) -> anyhow::Result<usize> {
|
||||||
|
let parsed = value
|
||||||
|
.parse::<usize>()
|
||||||
|
.map_err(|error| anyhow::anyhow!("{source} 无效:{error}"))?;
|
||||||
|
if !(MIN_DOWNLOAD_CONCURRENCY..=MAX_DOWNLOAD_CONCURRENCY).contains(&parsed) {
|
||||||
|
return Err(anyhow::anyhow!(
|
||||||
|
"{source} 必须在 {MIN_DOWNLOAD_CONCURRENCY}..={MAX_DOWNLOAD_CONCURRENCY} 范围内"
|
||||||
|
));
|
||||||
|
}
|
||||||
|
Ok(parsed)
|
||||||
|
}
|
||||||
|
|
||||||
fn next_option_value(
|
fn next_option_value(
|
||||||
args: &mut impl Iterator<Item = String>,
|
args: &mut impl Iterator<Item = String>,
|
||||||
flag: &str,
|
flag: &str,
|
||||||
@@ -7896,6 +8167,7 @@ fn print_usage(binary: &str) {
|
|||||||
eprintln!(" parse-text-units Query current official TextUnit detail index");
|
eprintln!(" parse-text-units Query current official TextUnit detail index");
|
||||||
eprintln!(" parse-errors Query current official parse/extraction diagnostics");
|
eprintln!(" parse-errors Query current official parse/extraction diagnostics");
|
||||||
eprintln!(" translation-tasks Query current offline TextUnit translation task status");
|
eprintln!(" translation-tasks Query current offline TextUnit translation task status");
|
||||||
|
eprintln!(" translation-handoff Query current translation job/unit/provider handoff");
|
||||||
eprintln!(" localized-status Show localized release status for current official release");
|
eprintln!(" localized-status Show localized release status for current official release");
|
||||||
eprintln!(" resource-index Query CAS + ResourceRepository index");
|
eprintln!(" resource-index Query CAS + ResourceRepository index");
|
||||||
eprintln!(" patch-apply Apply a Binary/JSON/Text patch file");
|
eprintln!(" patch-apply Apply a Binary/JSON/Text patch file");
|
||||||
@@ -7946,6 +8218,9 @@ fn print_usage(binary: &str) {
|
|||||||
eprintln!(" --import-resource-db <PATH> SQLite ResourceRepository path");
|
eprintln!(" --import-resource-db <PATH> SQLite ResourceRepository path");
|
||||||
eprintln!(" --snapshot <PATH> Override snapshot path (default: <output>/current/official-sync-snapshot.json)");
|
eprintln!(" --snapshot <PATH> Override snapshot path (default: <output>/current/official-sync-snapshot.json)");
|
||||||
eprintln!(" --curl <PATH> curl executable (default: curl)");
|
eprintln!(" --curl <PATH> curl executable (default: curl)");
|
||||||
|
eprintln!(
|
||||||
|
" --download-concurrency <N> Bounded parallel downloads (default: 8, range 1..=256)"
|
||||||
|
);
|
||||||
eprintln!(" --proxy <URL|auto|none> curl proxy override (default: auto from env)");
|
eprintln!(" --proxy <URL|auto|none> curl proxy override (default: auto from env)");
|
||||||
eprintln!(" --no-proxy Force direct curl connections");
|
eprintln!(" --no-proxy Force direct curl connections");
|
||||||
eprintln!(" --unzip <PATH> unzip executable (default: unzip)");
|
eprintln!(" --unzip <PATH> unzip executable (default: unzip)");
|
||||||
@@ -7970,6 +8245,9 @@ fn print_usage(binary: &str) {
|
|||||||
eprintln!(" --bundle-path <PATH> Filter resource-index by metadata bundle path");
|
eprintln!(" --bundle-path <PATH> Filter resource-index by metadata bundle path");
|
||||||
eprintln!(" --archive-entry <PATH> Filter resource-index, parse detail, or translation-tasks by ZIP/archive entry");
|
eprintln!(" --archive-entry <PATH> Filter resource-index, parse detail, or translation-tasks by ZIP/archive entry");
|
||||||
eprintln!(" --task-status <STATUS> Filter translation-tasks by task status");
|
eprintln!(" --task-status <STATUS> Filter translation-tasks by task status");
|
||||||
|
eprintln!(
|
||||||
|
" --worker-status <STATUS> Filter translation-tasks by provider worker status"
|
||||||
|
);
|
||||||
eprintln!(" --parse-status <STATUS> Filter resource-index or translation-tasks by parse status");
|
eprintln!(" --parse-status <STATUS> Filter resource-index or translation-tasks by parse status");
|
||||||
eprintln!(" --path-id <ID> Filter parse detail by Unity object path ID");
|
eprintln!(" --path-id <ID> Filter parse detail by Unity object path ID");
|
||||||
eprintln!(" --class-id <ID> Filter parse detail by Unity class ID");
|
eprintln!(" --class-id <ID> Filter parse detail by Unity class ID");
|
||||||
@@ -7978,6 +8256,9 @@ fn print_usage(binary: &str) {
|
|||||||
eprintln!(
|
eprintln!(
|
||||||
" --has-reason | --no-reason Filter translation-tasks by diagnostic reason presence"
|
" --has-reason | --no-reason Filter translation-tasks by diagnostic reason presence"
|
||||||
);
|
);
|
||||||
|
eprintln!(
|
||||||
|
" --has-failure-reason | --no-failure-reason Filter translation-tasks by provider failure reason"
|
||||||
|
);
|
||||||
eprintln!();
|
eprintln!();
|
||||||
eprintln!("Write patch:");
|
eprintln!("Write patch:");
|
||||||
eprintln!(" --patch-kind <binary|json|text> Patch type for patch-apply");
|
eprintln!(" --patch-kind <binary|json|text> Patch type for patch-apply");
|
||||||
@@ -8098,6 +8379,10 @@ mod tests {
|
|||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn env_defaults_apply_and_cli_overrides() {
|
fn env_defaults_apply_and_cli_overrides() {
|
||||||
|
assert_eq!(
|
||||||
|
CliOptions::default().config.download_concurrency,
|
||||||
|
DEFAULT_DOWNLOAD_CONCURRENCY
|
||||||
|
);
|
||||||
let options = parse_with_env(
|
let options = parse_with_env(
|
||||||
&["bat"],
|
&["bat"],
|
||||||
&[
|
&[
|
||||||
@@ -8109,6 +8394,7 @@ mod tests {
|
|||||||
("BAT_AUTO_DISCOVER", "1"),
|
("BAT_AUTO_DISCOVER", "1"),
|
||||||
("BAT_STATE_DIR", "/srv/state"),
|
("BAT_STATE_DIR", "/srv/state"),
|
||||||
("BAT_INTERVAL_SECONDS", "120"),
|
("BAT_INTERVAL_SECONDS", "120"),
|
||||||
|
("BAT_DOWNLOAD_CONCURRENCY", "3"),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
.unwrap();
|
.unwrap();
|
||||||
@@ -8127,6 +8413,7 @@ mod tests {
|
|||||||
Some(PathBuf::from("/srv/bat/resources.sqlite"))
|
Some(PathBuf::from("/srv/bat/resources.sqlite"))
|
||||||
);
|
);
|
||||||
assert!(options.config.auto_discover);
|
assert!(options.config.auto_discover);
|
||||||
|
assert_eq!(options.config.download_concurrency, 3);
|
||||||
assert_eq!(options.state_dir, PathBuf::from("/srv/state"));
|
assert_eq!(options.state_dir, PathBuf::from("/srv/state"));
|
||||||
assert_eq!(options.interval, Duration::from_secs(120));
|
assert_eq!(options.interval, Duration::from_secs(120));
|
||||||
|
|
||||||
@@ -8206,6 +8493,20 @@ mod tests {
|
|||||||
assert!(parse_with_env(&["bat"], &[("BAT_WATCH", "maybe")]).is_err());
|
assert!(parse_with_env(&["bat"], &[("BAT_WATCH", "maybe")]).is_err());
|
||||||
assert!(parse_with_env(&["bat"], &[("BAT_INTERVAL_SECONDS", "abc")]).is_err());
|
assert!(parse_with_env(&["bat"], &[("BAT_INTERVAL_SECONDS", "abc")]).is_err());
|
||||||
assert!(parse_with_env(&["bat"], &[("BAT_PROXY", "ftp://x")]).is_err());
|
assert!(parse_with_env(&["bat"], &[("BAT_PROXY", "ftp://x")]).is_err());
|
||||||
|
assert!(parse_with_env(&["bat"], &[("BAT_DOWNLOAD_CONCURRENCY", "0")]).is_err());
|
||||||
|
assert!(parse_with_env(&["bat"], &[("BAT_DOWNLOAD_CONCURRENCY", "257")]).is_err());
|
||||||
|
assert!(parse(&["bat", "--download-concurrency", "0"]).is_err());
|
||||||
|
assert!(parse(&["bat", "--download-concurrency", "257"]).is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn cli_download_concurrency_is_preserved_for_daemon_child() {
|
||||||
|
let options = parse(&["bat", "--download-concurrency", "4"]).unwrap();
|
||||||
|
assert_eq!(options.config.download_concurrency, 4);
|
||||||
|
let args = daemon_child_args(&options);
|
||||||
|
assert!(args
|
||||||
|
.windows(2)
|
||||||
|
.any(|pair| pair == ["--download-concurrency", "4"]));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -11242,6 +11543,34 @@ mod tests {
|
|||||||
bat_infrastructure::write_textunit_task_queue_at(current_dir, &queue).unwrap();
|
bat_infrastructure::write_textunit_task_queue_at(current_dir, &queue).unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn write_translation_task_repository_fixture(current_dir: &Path) {
|
||||||
|
let queue = bat_infrastructure::read_textunit_task_queue_at(current_dir)
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
let repository_path =
|
||||||
|
bat_infrastructure::SqliteTranslationTaskRepository::repository_path(current_dir);
|
||||||
|
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||||
|
.enable_all()
|
||||||
|
.build()
|
||||||
|
.unwrap();
|
||||||
|
runtime.block_on(async {
|
||||||
|
let repository =
|
||||||
|
bat_infrastructure::SqliteTranslationTaskRepository::new(&repository_path)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
repository.sync_queue(&queue).await.unwrap();
|
||||||
|
repository
|
||||||
|
.update_status(
|
||||||
|
"textunit/v-current/Bundle/a.bundle",
|
||||||
|
bat_infrastructure::TranslationTaskStatus::Failed,
|
||||||
|
Some("Crowdin provider rejected the payload".to_string()),
|
||||||
|
Some("crowdin-run-1".to_string()),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
fn write_textunit_index_fixture(current_dir: &Path) {
|
fn write_textunit_index_fixture(current_dir: &Path) {
|
||||||
let index = bat_infrastructure::OfficialTextUnitIndex {
|
let index = bat_infrastructure::OfficialTextUnitIndex {
|
||||||
version: bat_infrastructure::OFFICIAL_TEXTUNIT_INDEX_VERSION,
|
version: bat_infrastructure::OFFICIAL_TEXTUNIT_INDEX_VERSION,
|
||||||
@@ -11326,6 +11655,7 @@ mod tests {
|
|||||||
let output_root = temp.path().join("output");
|
let output_root = temp.path().join("output");
|
||||||
let current_dir = write_catalog_fixture(&state_dir, &output_root, "bundle-b2", None);
|
let current_dir = write_catalog_fixture(&state_dir, &output_root, "bundle-b2", None);
|
||||||
write_textunit_task_queue_fixture(¤t_dir);
|
write_textunit_task_queue_fixture(¤t_dir);
|
||||||
|
write_translation_task_repository_fixture(¤t_dir);
|
||||||
|
|
||||||
let envelope = dispatch_rpc_method(
|
let envelope = dispatch_rpc_method(
|
||||||
&rpc_request(
|
&rpc_request(
|
||||||
@@ -11392,6 +11722,80 @@ mod tests {
|
|||||||
value["data"]["entries"][0]["reason"],
|
value["data"]["entries"][0]["reason"],
|
||||||
serde_json::Value::Null
|
serde_json::Value::Null
|
||||||
);
|
);
|
||||||
|
|
||||||
|
let envelope = dispatch_rpc_method(
|
||||||
|
&rpc_request(
|
||||||
|
"translation.tasks",
|
||||||
|
Some(serde_json::json!({
|
||||||
|
"worker_status": "failed",
|
||||||
|
"has_failure_reason": true,
|
||||||
|
"offset": 0,
|
||||||
|
"limit": 10
|
||||||
|
})),
|
||||||
|
),
|
||||||
|
&state_dir,
|
||||||
|
&new_daemon_control(),
|
||||||
|
&test_task_context(),
|
||||||
|
"req-translation-3".to_string(),
|
||||||
|
);
|
||||||
|
let value = serde_json::to_value(&envelope).unwrap();
|
||||||
|
assert_eq!(value["ok"], true);
|
||||||
|
assert_eq!(value["data"]["task_repository_available"], true);
|
||||||
|
assert_eq!(value["data"]["total_entries"], 1);
|
||||||
|
assert_eq!(value["data"]["entries"][0]["task_status"], "failed");
|
||||||
|
assert_eq!(
|
||||||
|
value["data"]["entries"][0]["failure_reason"],
|
||||||
|
"Crowdin provider rejected the payload"
|
||||||
|
);
|
||||||
|
assert_eq!(value["data"]["entries"][0]["attempt_count"], 0);
|
||||||
|
|
||||||
|
let envelope = dispatch_rpc_method(
|
||||||
|
&rpc_request(
|
||||||
|
"translation.task.update",
|
||||||
|
Some(serde_json::json!({
|
||||||
|
"task_id": "textunit/v-current/Bundle/a.bundle",
|
||||||
|
"status": "completed",
|
||||||
|
"provider_run_id": "crowdin-run-2"
|
||||||
|
})),
|
||||||
|
),
|
||||||
|
&state_dir,
|
||||||
|
&new_daemon_control(),
|
||||||
|
&test_task_context(),
|
||||||
|
"req-translation-4".to_string(),
|
||||||
|
);
|
||||||
|
let value = serde_json::to_value(&envelope).unwrap();
|
||||||
|
assert_eq!(value["ok"], true);
|
||||||
|
assert_eq!(value["data"]["entry"]["task_status"], "completed");
|
||||||
|
assert_eq!(
|
||||||
|
value["data"]["entry"]["failure_reason"],
|
||||||
|
serde_json::Value::Null
|
||||||
|
);
|
||||||
|
assert_eq!(value["data"]["entry"]["provider_run_id"], "crowdin-run-2");
|
||||||
|
assert!(value["data"]["entry"]["completed_unix_seconds"].is_number());
|
||||||
|
|
||||||
|
let envelope = dispatch_rpc_method(
|
||||||
|
&rpc_request("translation.handoff", None),
|
||||||
|
&state_dir,
|
||||||
|
&new_daemon_control(),
|
||||||
|
&test_task_context(),
|
||||||
|
"req-translation-handoff".to_string(),
|
||||||
|
);
|
||||||
|
let value = serde_json::to_value(&envelope).unwrap();
|
||||||
|
assert_eq!(value["ok"], true);
|
||||||
|
assert_eq!(value["data"]["available"], true);
|
||||||
|
assert_eq!(
|
||||||
|
value["data"]["handoff_schema_version"],
|
||||||
|
bat_infrastructure::TRANSLATION_HANDOFF_SCHEMA_VERSION
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
value["data"]["handoff"]["units"].as_array().unwrap().len(),
|
||||||
|
2
|
||||||
|
);
|
||||||
|
assert_eq!(value["data"]["handoff"]["units"][0]["status"], "translated");
|
||||||
|
assert_eq!(
|
||||||
|
value["data"]["handoff"]["provider_runs"][0]["provider_run_id"],
|
||||||
|
"crowdin-run-2"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
@@ -0,0 +1,260 @@
|
|||||||
|
//! Downloader backend and bounded scheduling contracts.
|
||||||
|
//!
|
||||||
|
//! The scheduler is deliberately independent from curl, manifests, and
|
||||||
|
//! official URL rules. Those concerns belong to a backend and the caller,
|
||||||
|
//! which keeps retry, proxy, and verification policy composable.
|
||||||
|
|
||||||
|
use std::sync::{mpsc, Arc, Mutex};
|
||||||
|
|
||||||
|
/// Lowest supported download concurrency.
|
||||||
|
pub const MIN_DOWNLOAD_CONCURRENCY: usize = 1;
|
||||||
|
/// Highest supported download concurrency.
|
||||||
|
pub const MAX_DOWNLOAD_CONCURRENCY: usize = 256;
|
||||||
|
/// Default official download concurrency.
|
||||||
|
pub const DEFAULT_DOWNLOAD_CONCURRENCY: usize = 8;
|
||||||
|
|
||||||
|
/// A backend that executes one already-planned download task.
|
||||||
|
pub trait DownloaderBackend<T>: Send + Sync {
|
||||||
|
/// Successful result returned for one task.
|
||||||
|
type Output: Send;
|
||||||
|
/// Failure returned for one task.
|
||||||
|
type Error: Send;
|
||||||
|
|
||||||
|
/// Executes one task. The scheduler owns ordering and concurrency only.
|
||||||
|
fn download(&self, task: T) -> Result<Self::Output, Self::Error>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A bounded worker scheduler.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub struct DownloadScheduler {
|
||||||
|
max_concurrency: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DownloadScheduler {
|
||||||
|
/// Creates a scheduler with the supported bounded range.
|
||||||
|
///
|
||||||
|
/// The official CLI validates input and reports out-of-range values.
|
||||||
|
/// This lower-level constructor remains total for library callers and
|
||||||
|
/// clamps values to the same safety bounds.
|
||||||
|
pub fn new(max_concurrency: usize) -> Self {
|
||||||
|
Self {
|
||||||
|
max_concurrency: max_concurrency
|
||||||
|
.clamp(MIN_DOWNLOAD_CONCURRENCY, MAX_DOWNLOAD_CONCURRENCY),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured upper bound.
|
||||||
|
pub fn max_concurrency(self) -> usize {
|
||||||
|
self.max_concurrency
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Executes tasks with a bounded number of workers.
|
||||||
|
///
|
||||||
|
/// Results are returned in input order even when workers finish out of
|
||||||
|
/// order. A failed task does not cause additional tasks to be scheduled
|
||||||
|
/// after it, because already-started bounded work must be joined cleanly;
|
||||||
|
/// callers decide whether a failed result invalidates the whole release.
|
||||||
|
pub fn execute<T, B>(self, backend: &B, tasks: Vec<T>) -> Vec<Result<B::Output, B::Error>>
|
||||||
|
where
|
||||||
|
T: Send + 'static,
|
||||||
|
B: DownloaderBackend<T>,
|
||||||
|
{
|
||||||
|
self.execute_with_observer(
|
||||||
|
backend,
|
||||||
|
tasks,
|
||||||
|
|_, _| Ok::<(), std::convert::Infallible>(()),
|
||||||
|
)
|
||||||
|
.expect("infallible download observer cannot fail")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Executes tasks and observes each result as soon as a worker returns it.
|
||||||
|
///
|
||||||
|
/// The observer runs on the coordinator thread, while worker threads
|
||||||
|
/// immediately take another pending task after sending their result. An
|
||||||
|
/// observer error stops further observation but still drains and joins all
|
||||||
|
/// workers before returning, so no background transfer is left detached.
|
||||||
|
pub fn execute_with_observer<T, B, F, E>(
|
||||||
|
self,
|
||||||
|
backend: &B,
|
||||||
|
tasks: Vec<T>,
|
||||||
|
mut observer: F,
|
||||||
|
) -> Result<Vec<Result<B::Output, B::Error>>, E>
|
||||||
|
where
|
||||||
|
T: Send + 'static,
|
||||||
|
B: DownloaderBackend<T>,
|
||||||
|
F: FnMut(usize, &Result<B::Output, B::Error>) -> Result<(), E>,
|
||||||
|
{
|
||||||
|
if tasks.is_empty() {
|
||||||
|
return Ok(Vec::new());
|
||||||
|
}
|
||||||
|
|
||||||
|
if self.max_concurrency == 1 {
|
||||||
|
let mut results = Vec::with_capacity(tasks.len());
|
||||||
|
for (index, task) in tasks.into_iter().enumerate() {
|
||||||
|
let result = backend.download(task);
|
||||||
|
observer(index, &result)?;
|
||||||
|
results.push(result);
|
||||||
|
}
|
||||||
|
return Ok(results);
|
||||||
|
}
|
||||||
|
|
||||||
|
let total = tasks.len();
|
||||||
|
let worker_count = self.max_concurrency.min(total);
|
||||||
|
let pending = Arc::new(Mutex::new(tasks.into_iter().enumerate()));
|
||||||
|
let (result_sender, result_receiver) = mpsc::channel();
|
||||||
|
|
||||||
|
std::thread::scope(|scope| {
|
||||||
|
for _ in 0..worker_count {
|
||||||
|
let pending = Arc::clone(&pending);
|
||||||
|
let result_sender = result_sender.clone();
|
||||||
|
scope.spawn(move || loop {
|
||||||
|
let task = pending
|
||||||
|
.lock()
|
||||||
|
.expect("download scheduler task queue poisoned")
|
||||||
|
.next();
|
||||||
|
let Some((index, task)) = task else {
|
||||||
|
break;
|
||||||
|
};
|
||||||
|
let result = backend.download(task);
|
||||||
|
if result_sender.send((index, result)).is_err() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
drop(result_sender);
|
||||||
|
|
||||||
|
let mut results = std::iter::repeat_with(|| None)
|
||||||
|
.take(total)
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
let mut observer_error = None;
|
||||||
|
for (index, result) in result_receiver {
|
||||||
|
if observer_error.is_none() {
|
||||||
|
if let Err(error) = observer(index, &result) {
|
||||||
|
observer_error = Some(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
results[index] = Some(result);
|
||||||
|
}
|
||||||
|
let results = results
|
||||||
|
.into_iter()
|
||||||
|
.map(|result| result.expect("download scheduler lost a task result"))
|
||||||
|
.collect();
|
||||||
|
match observer_error {
|
||||||
|
Some(error) => Err(error),
|
||||||
|
None => Ok(results),
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||||
|
use std::thread;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
struct TestBackend {
|
||||||
|
active: AtomicUsize,
|
||||||
|
max_active: AtomicUsize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DownloaderBackend<usize> for TestBackend {
|
||||||
|
type Output = usize;
|
||||||
|
type Error = String;
|
||||||
|
|
||||||
|
fn download(&self, task: usize) -> Result<Self::Output, Self::Error> {
|
||||||
|
let active = self.active.fetch_add(1, Ordering::SeqCst) + 1;
|
||||||
|
self.max_active.fetch_max(active, Ordering::SeqCst);
|
||||||
|
thread::sleep(Duration::from_millis(2));
|
||||||
|
self.active.fetch_sub(1, Ordering::SeqCst);
|
||||||
|
Ok(task * 2)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn scheduler_preserves_result_order_and_respects_bound() {
|
||||||
|
let backend = TestBackend {
|
||||||
|
active: AtomicUsize::new(0),
|
||||||
|
max_active: AtomicUsize::new(0),
|
||||||
|
};
|
||||||
|
let results = DownloadScheduler::new(2).execute(&backend, (0..8).collect());
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
results.into_iter().map(Result::unwrap).collect::<Vec<_>>(),
|
||||||
|
(0..8).map(|value| value * 2).collect::<Vec<_>>()
|
||||||
|
);
|
||||||
|
assert!(backend.max_active.load(Ordering::SeqCst) <= 2);
|
||||||
|
assert!(backend.max_active.load(Ordering::SeqCst) >= 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn zero_concurrency_is_conservative() {
|
||||||
|
assert_eq!(
|
||||||
|
DownloadScheduler::new(0).max_concurrency(),
|
||||||
|
MIN_DOWNLOAD_CONCURRENCY
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn scheduler_caps_untrusted_upper_bound() {
|
||||||
|
assert_eq!(
|
||||||
|
DownloadScheduler::new(usize::MAX).max_concurrency(),
|
||||||
|
MAX_DOWNLOAD_CONCURRENCY
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn observer_receives_completion_without_a_global_barrier() {
|
||||||
|
struct UnevenBackend {
|
||||||
|
active: AtomicUsize,
|
||||||
|
task_two_started_while_task_zero_active: AtomicUsize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DownloaderBackend<usize> for UnevenBackend {
|
||||||
|
type Output = usize;
|
||||||
|
type Error = String;
|
||||||
|
|
||||||
|
fn download(&self, task: usize) -> Result<Self::Output, Self::Error> {
|
||||||
|
if task == 0 {
|
||||||
|
self.active.fetch_add(1, Ordering::SeqCst);
|
||||||
|
thread::sleep(Duration::from_millis(20));
|
||||||
|
self.active.fetch_sub(1, Ordering::SeqCst);
|
||||||
|
} else {
|
||||||
|
if task == 2 && self.active.load(Ordering::SeqCst) > 0 {
|
||||||
|
self.task_two_started_while_task_zero_active
|
||||||
|
.fetch_add(1, Ordering::SeqCst);
|
||||||
|
}
|
||||||
|
thread::sleep(Duration::from_millis(if task == 1 { 1 } else { 5 }));
|
||||||
|
}
|
||||||
|
Ok(task)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let backend = UnevenBackend {
|
||||||
|
active: AtomicUsize::new(0),
|
||||||
|
task_two_started_while_task_zero_active: AtomicUsize::new(0),
|
||||||
|
};
|
||||||
|
let mut completed = Vec::new();
|
||||||
|
let results = DownloadScheduler::new(2)
|
||||||
|
.execute_with_observer(&backend, vec![0, 1, 2], |index, _| {
|
||||||
|
completed.push(index);
|
||||||
|
Ok::<(), ()>(())
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
results.into_iter().map(Result::unwrap).collect::<Vec<_>>(),
|
||||||
|
vec![0, 1, 2]
|
||||||
|
);
|
||||||
|
assert_eq!(completed.len(), 3);
|
||||||
|
assert!(completed[0] == 1, "短任务应在长任务之前回传:{completed:?}");
|
||||||
|
assert_eq!(
|
||||||
|
backend
|
||||||
|
.task_two_started_while_task_zero_active
|
||||||
|
.load(Ordering::SeqCst),
|
||||||
|
1,
|
||||||
|
"worker 完成 task 1 后应立即领取 task 2"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -12,6 +12,7 @@
|
|||||||
|
|
||||||
pub mod cas;
|
pub mod cas;
|
||||||
mod curl_transfer;
|
mod curl_transfer;
|
||||||
|
pub mod downloader;
|
||||||
pub mod import;
|
pub mod import;
|
||||||
pub mod localized_patch;
|
pub mod localized_patch;
|
||||||
pub mod official_changes;
|
pub mod official_changes;
|
||||||
@@ -28,12 +29,17 @@ pub mod patch_ops;
|
|||||||
pub mod path_security;
|
pub mod path_security;
|
||||||
pub mod release_flow;
|
pub mod release_flow;
|
||||||
pub mod resources;
|
pub mod resources;
|
||||||
|
pub mod translation_tasks;
|
||||||
mod zip_validation;
|
mod zip_validation;
|
||||||
|
|
||||||
pub use cas::FileSystemCasRepository;
|
pub use cas::FileSystemCasRepository;
|
||||||
pub use curl_transfer::{
|
pub use curl_transfer::{
|
||||||
redact_proxy_url, resolve_curl_proxy, CurlProxyConfig, CurlProxyMode, ResolvedCurlProxy,
|
redact_proxy_url, resolve_curl_proxy, CurlProxyConfig, CurlProxyMode, ResolvedCurlProxy,
|
||||||
};
|
};
|
||||||
|
pub use downloader::{
|
||||||
|
DownloadScheduler, DownloaderBackend, DEFAULT_DOWNLOAD_CONCURRENCY, MAX_DOWNLOAD_CONCURRENCY,
|
||||||
|
MIN_DOWNLOAD_CONCURRENCY,
|
||||||
|
};
|
||||||
pub use import::{
|
pub use import::{
|
||||||
BundleSource, ImportedResource, ResourceImportCategory, ResourceImportReport,
|
BundleSource, ImportedResource, ResourceImportCategory, ResourceImportReport,
|
||||||
ResourceImportService,
|
ResourceImportService,
|
||||||
@@ -124,6 +130,14 @@ pub use path_security::{
|
|||||||
};
|
};
|
||||||
pub use release_flow::ReleaseFlowStatusCode;
|
pub use release_flow::ReleaseFlowStatusCode;
|
||||||
pub use resources::{InMemoryResourceRepository, SqliteResourceRepository};
|
pub use resources::{InMemoryResourceRepository, SqliteResourceRepository};
|
||||||
|
pub use translation_tasks::{
|
||||||
|
build_translation_handoff, read_translation_handoff_at, sync_translation_task_repository_at,
|
||||||
|
write_translation_handoff_at, PersistedTranslationTask, ProviderRun, ProviderRunStatus,
|
||||||
|
SqliteTranslationTaskRepository, TranslationHandoff, TranslationJob, TranslationJobStatus,
|
||||||
|
TranslationTaskStatus, TranslationTaskSyncReport, TranslationUnit, TranslationUnitStatus,
|
||||||
|
TRANSLATION_HANDOFF_FILE, TRANSLATION_HANDOFF_SCHEMA_VERSION, TRANSLATION_TASK_REPOSITORY_FILE,
|
||||||
|
TRANSLATION_TASK_SCHEMA_VERSION,
|
||||||
|
};
|
||||||
|
|
||||||
/// Infrastructure 版本号
|
/// Infrastructure 版本号
|
||||||
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
|
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
|
||||||
|
|||||||
@@ -1,6 +1,10 @@
|
|||||||
//! Official JP resource download execution.
|
//! Official JP resource download execution.
|
||||||
|
|
||||||
use crate::curl_transfer::{run_curl_with_retry_with_proxy, CurlProxyConfig, CurlRetryError};
|
use crate::curl_transfer::{run_curl_with_retry_with_proxy, CurlProxyConfig, CurlRetryError};
|
||||||
|
use crate::downloader::{
|
||||||
|
DownloadScheduler, DownloaderBackend, DEFAULT_DOWNLOAD_CONCURRENCY, MAX_DOWNLOAD_CONCURRENCY,
|
||||||
|
MIN_DOWNLOAD_CONCURRENCY,
|
||||||
|
};
|
||||||
use crate::official_pull::OfficialResourcePullPlan;
|
use crate::official_pull::OfficialResourcePullPlan;
|
||||||
use crate::path_security::{
|
use crate::path_security::{
|
||||||
ensure_path_within_root, ensure_safe_directory_path, ensure_safe_file_target,
|
ensure_path_within_root, ensure_safe_directory_path, ensure_safe_file_target,
|
||||||
@@ -9,7 +13,10 @@ use crate::path_security::{
|
|||||||
use crate::zip_validation::{
|
use crate::zip_validation::{
|
||||||
path_has_zip_extension, url_or_path_has_zip_extension, validate_zip_structure,
|
path_has_zip_extension, url_or_path_has_zip_extension, validate_zip_structure,
|
||||||
};
|
};
|
||||||
use bat_adapters::official::yostar_jp::{is_official_yostar_jp_url, YostarJpResourceEndpointKind};
|
use bat_adapters::official::yostar_jp::YostarJpResourceEndpointKind;
|
||||||
|
use bat_adapters::official::{
|
||||||
|
destination_under_root, DownloadUrlMapper, OfficialResourceBackend, YostarJpBackend,
|
||||||
|
};
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use std::collections::{BTreeMap, HashSet};
|
use std::collections::{BTreeMap, HashSet};
|
||||||
use std::fs::{self, File};
|
use std::fs::{self, File};
|
||||||
@@ -539,9 +546,11 @@ impl OfficialLocalResourceState {
|
|||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct OfficialResourcePullService {
|
pub struct OfficialResourcePullService {
|
||||||
output_root: PathBuf,
|
output_root: PathBuf,
|
||||||
|
backend: YostarJpBackend,
|
||||||
curl_command: PathBuf,
|
curl_command: PathBuf,
|
||||||
curl_proxy: CurlProxyConfig,
|
curl_proxy: CurlProxyConfig,
|
||||||
retry_attempts: usize,
|
retry_attempts: usize,
|
||||||
|
max_concurrency: usize,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl OfficialResourcePullService {
|
impl OfficialResourcePullService {
|
||||||
@@ -557,9 +566,11 @@ impl OfficialResourcePullService {
|
|||||||
) -> Self {
|
) -> Self {
|
||||||
Self {
|
Self {
|
||||||
output_root: output_root.into(),
|
output_root: output_root.into(),
|
||||||
|
backend: YostarJpBackend,
|
||||||
curl_command: curl_command.into(),
|
curl_command: curl_command.into(),
|
||||||
curl_proxy: CurlProxyConfig::default(),
|
curl_proxy: CurlProxyConfig::default(),
|
||||||
retry_attempts: DEFAULT_RETRY_ATTEMPTS,
|
retry_attempts: DEFAULT_RETRY_ATTEMPTS,
|
||||||
|
max_concurrency: DEFAULT_DOWNLOAD_CONCURRENCY,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -575,6 +586,22 @@ impl OfficialResourcePullService {
|
|||||||
self
|
self
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Sets the bounded number of concurrent downloads.
|
||||||
|
///
|
||||||
|
/// The default is [`DEFAULT_DOWNLOAD_CONCURRENCY`]. Values are kept
|
||||||
|
/// within the supported `1..=256` range; the public update configuration
|
||||||
|
/// validates input before constructing this service.
|
||||||
|
pub fn with_max_concurrency(mut self, max_concurrency: usize) -> Self {
|
||||||
|
self.max_concurrency =
|
||||||
|
max_concurrency.clamp(MIN_DOWNLOAD_CONCURRENCY, MAX_DOWNLOAD_CONCURRENCY);
|
||||||
|
self
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured download concurrency.
|
||||||
|
pub fn max_concurrency(&self) -> usize {
|
||||||
|
self.max_concurrency
|
||||||
|
}
|
||||||
|
|
||||||
/// Returns the output root used for downloaded files.
|
/// Returns the output root used for downloaded files.
|
||||||
pub fn output_root(&self) -> &Path {
|
pub fn output_root(&self) -> &Path {
|
||||||
&self.output_root
|
&self.output_root
|
||||||
@@ -666,7 +693,7 @@ impl OfficialResourcePullService {
|
|||||||
}
|
}
|
||||||
let mut planned: Vec<PlannedDownload> = Vec::with_capacity(total);
|
let mut planned: Vec<PlannedDownload> = Vec::with_capacity(total);
|
||||||
for url in urls {
|
for url in urls {
|
||||||
if !is_official_yostar_jp_url(&url) {
|
if !self.backend.is_official_url(&url) {
|
||||||
return Err(DownloadError::new(
|
return Err(DownloadError::new(
|
||||||
bat_core::ErrorCode::NON_OFFICIAL_URL,
|
bat_core::ErrorCode::NON_OFFICIAL_URL,
|
||||||
format!("拒绝下载非官方 URL:{url}"),
|
format!("拒绝下载非官方 URL:{url}"),
|
||||||
@@ -695,6 +722,17 @@ impl OfficialResourcePullService {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if self.max_concurrency > 1 {
|
||||||
|
return self.pull_planned_concurrently(
|
||||||
|
planned,
|
||||||
|
manifest,
|
||||||
|
official_hash_pairs,
|
||||||
|
total,
|
||||||
|
&mut progress,
|
||||||
|
&mut should_cancel,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
// Phase B:按 plan 顺序处理每个 URL。下载或复用完成并写入 manifest 后,
|
// Phase B:按 plan 顺序处理每个 URL。下载或复用完成并写入 manifest 后,
|
||||||
// 立即尝试校验已经到齐的官方 `.bytes/.hash` pair,避免把文件级问题延后到整轮末尾。
|
// 立即尝试校验已经到齐的官方 `.bytes/.hash` pair,避免把文件级问题延后到整轮末尾。
|
||||||
let mut completed_count = 0usize;
|
let mut completed_count = 0usize;
|
||||||
@@ -824,6 +862,183 @@ impl OfficialResourcePullService {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn pull_planned_concurrently(
|
||||||
|
&self,
|
||||||
|
planned: Vec<PlannedDownload>,
|
||||||
|
mut manifest: OfficialDownloadManifest,
|
||||||
|
official_hash_pairs: Vec<OfficialSeedHashPair>,
|
||||||
|
total: usize,
|
||||||
|
progress: &mut impl FnMut(OfficialResourcePullProgress),
|
||||||
|
should_cancel: &mut impl FnMut() -> bool,
|
||||||
|
) -> Result<OfficialResourcePullReport, DownloadError> {
|
||||||
|
if should_cancel() {
|
||||||
|
return Err("官方资源拉取已被停止请求中断".to_string().into());
|
||||||
|
}
|
||||||
|
for item in &planned {
|
||||||
|
progress(OfficialResourcePullProgress::started(
|
||||||
|
0,
|
||||||
|
total,
|
||||||
|
item.url.clone(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
let backend = CurlDownloadBackend { service: self };
|
||||||
|
let mut completed_count = 0usize;
|
||||||
|
let mut items_by_plan_index: Vec<Option<OfficialResourcePullItem>> =
|
||||||
|
(0..total).map(|_| None).collect();
|
||||||
|
let mut verified_hashes = Vec::new();
|
||||||
|
let mut verified_hash_urls = HashSet::<String>::new();
|
||||||
|
let mut processed_urls = HashSet::<String>::new();
|
||||||
|
|
||||||
|
DownloadScheduler::new(self.max_concurrency).execute_with_observer(
|
||||||
|
&backend,
|
||||||
|
planned.clone(),
|
||||||
|
|plan_index, result| {
|
||||||
|
let item = &planned[plan_index];
|
||||||
|
if should_cancel() {
|
||||||
|
return Err("官方资源拉取已被停止请求中断".to_string().into());
|
||||||
|
}
|
||||||
|
|
||||||
|
let needs_manifest = item.existing.is_none();
|
||||||
|
let verification = match result {
|
||||||
|
Ok(pull_result) => {
|
||||||
|
let verification_result = self
|
||||||
|
.clear_quarantine_entry(&item.url)
|
||||||
|
.and_then(|_| {
|
||||||
|
if needs_manifest {
|
||||||
|
self.record_download_manifest_entry(
|
||||||
|
&mut manifest,
|
||||||
|
&item.url,
|
||||||
|
&item.destination,
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
Ok(pull_result.verification.clone())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.and_then(|verification| {
|
||||||
|
if needs_manifest {
|
||||||
|
self.write_download_manifest(&manifest)
|
||||||
|
.map(|_| verification)
|
||||||
|
} else {
|
||||||
|
Ok(verification)
|
||||||
|
}
|
||||||
|
});
|
||||||
|
match verification_result {
|
||||||
|
Ok(verification) => verification,
|
||||||
|
Err(error) => {
|
||||||
|
let error = PullOneError::plain(format!(
|
||||||
|
"记录下载 manifest 失败:{error}"
|
||||||
|
));
|
||||||
|
self.record_quarantine_entry(
|
||||||
|
&item.url,
|
||||||
|
&item.destination,
|
||||||
|
&error,
|
||||||
|
)?;
|
||||||
|
progress(OfficialResourcePullProgress::failed(
|
||||||
|
completed_count,
|
||||||
|
total,
|
||||||
|
item.url.clone(),
|
||||||
|
&error,
|
||||||
|
));
|
||||||
|
return Err(DownloadError::new(
|
||||||
|
error.error_code(),
|
||||||
|
format!(
|
||||||
|
"官方资源下载失败:URL 已进入 quarantine,中止本轮同步、不发布不完整资源;url={} quarantine={};{}",
|
||||||
|
item.url,
|
||||||
|
self.download_quarantine_path().display(),
|
||||||
|
error.message
|
||||||
|
),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(error) => {
|
||||||
|
self.record_quarantine_entry(&item.url, &item.destination, error)?;
|
||||||
|
progress(OfficialResourcePullProgress::failed(
|
||||||
|
completed_count,
|
||||||
|
total,
|
||||||
|
item.url.clone(),
|
||||||
|
error,
|
||||||
|
));
|
||||||
|
return Err(DownloadError::new(
|
||||||
|
error.error_code(),
|
||||||
|
format!(
|
||||||
|
"官方资源下载失败:URL 已进入 quarantine,中止本轮同步、不发布不完整资源;url={} quarantine={};{}",
|
||||||
|
item.url,
|
||||||
|
self.download_quarantine_path().display(),
|
||||||
|
error.message
|
||||||
|
),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
completed_count += 1;
|
||||||
|
progress(OfficialResourcePullProgress::finished(
|
||||||
|
completed_count,
|
||||||
|
total,
|
||||||
|
item.url.clone(),
|
||||||
|
result
|
||||||
|
.as_ref()
|
||||||
|
.expect("successful result handled above")
|
||||||
|
.status,
|
||||||
|
result
|
||||||
|
.as_ref()
|
||||||
|
.expect("successful result handled above")
|
||||||
|
.bytes,
|
||||||
|
result
|
||||||
|
.as_ref()
|
||||||
|
.expect("successful result handled above")
|
||||||
|
.transferred_bytes,
|
||||||
|
verification,
|
||||||
|
));
|
||||||
|
processed_urls.insert(item.url.clone());
|
||||||
|
let newly_verified_hashes = self.verify_ready_official_hashes(
|
||||||
|
&official_hash_pairs,
|
||||||
|
&processed_urls,
|
||||||
|
&mut verified_hash_urls,
|
||||||
|
&mut verified_hashes,
|
||||||
|
&mut manifest,
|
||||||
|
)?;
|
||||||
|
for verification in newly_verified_hashes {
|
||||||
|
progress(OfficialResourcePullProgress::verification(
|
||||||
|
completed_count,
|
||||||
|
total,
|
||||||
|
verification.data_url.clone(),
|
||||||
|
verification,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let pull_result = result
|
||||||
|
.as_ref()
|
||||||
|
.expect("successful result handled above");
|
||||||
|
items_by_plan_index[plan_index] = Some(OfficialResourcePullItem {
|
||||||
|
url: item.url.clone(),
|
||||||
|
destination: item.destination.clone(),
|
||||||
|
bytes: pull_result.bytes,
|
||||||
|
transferred_bytes: pull_result.transferred_bytes,
|
||||||
|
status: pull_result.status,
|
||||||
|
});
|
||||||
|
Ok(())
|
||||||
|
},
|
||||||
|
)?;
|
||||||
|
|
||||||
|
self.verify_all_official_hashes_are_complete(&official_hash_pairs, &verified_hash_urls)?;
|
||||||
|
let items = items_by_plan_index
|
||||||
|
.into_iter()
|
||||||
|
.enumerate()
|
||||||
|
.map(|(index, item)| {
|
||||||
|
item.ok_or_else(|| {
|
||||||
|
DownloadError::from(format!(
|
||||||
|
"并发下载结果缺少 plan index={index},拒绝发布不完整资源"
|
||||||
|
))
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect::<Result<Vec<_>, _>>()?;
|
||||||
|
Ok(OfficialResourcePullReport {
|
||||||
|
items,
|
||||||
|
verified_hashes,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/// Audits every URL in a pull plan against the local download manifest.
|
/// Audits every URL in a pull plan against the local download manifest.
|
||||||
///
|
///
|
||||||
/// This performs no network I/O. It checks that each URL has a manifest
|
/// This performs no network I/O. It checks that each URL has a manifest
|
||||||
@@ -914,7 +1129,7 @@ impl OfficialResourcePullService {
|
|||||||
/// `TableCatalog.bytes`, `BundlePackingInfo.bytes`, and
|
/// `TableCatalog.bytes`, `BundlePackingInfo.bytes`, and
|
||||||
/// `MediaCatalog.bytes`.
|
/// `MediaCatalog.bytes`.
|
||||||
pub fn fetch_bytes(&self, url: &str) -> Result<Vec<u8>, DownloadError> {
|
pub fn fetch_bytes(&self, url: &str) -> Result<Vec<u8>, DownloadError> {
|
||||||
if !is_official_yostar_jp_url(url) {
|
if !self.backend.is_official_url(url) {
|
||||||
return Err(DownloadError::new(
|
return Err(DownloadError::new(
|
||||||
bat_core::ErrorCode::NON_OFFICIAL_URL,
|
bat_core::ErrorCode::NON_OFFICIAL_URL,
|
||||||
format!("拒绝拉取非官方 URL:{url}"),
|
format!("拒绝拉取非官方 URL:{url}"),
|
||||||
@@ -1581,36 +1796,12 @@ impl OfficialResourcePullService {
|
|||||||
|
|
||||||
fn destination_for_url(&self, url: &str) -> Result<PathBuf, String> {
|
fn destination_for_url(&self, url: &str) -> Result<PathBuf, String> {
|
||||||
self.ensure_output_root_safe()?;
|
self.ensure_output_root_safe()?;
|
||||||
if !is_official_yostar_jp_url(url) {
|
if !self.backend.is_official_url(url) {
|
||||||
return Err(format!("URL 不是官方 JP host:{url}"));
|
return Err(format!("URL 不是官方 JP host:{url}"));
|
||||||
}
|
}
|
||||||
|
|
||||||
let rest = url
|
let relative_destination = self.backend.relative_destination(url)?;
|
||||||
.strip_prefix("https://")
|
let destination = destination_under_root(&self.output_root, &relative_destination)?;
|
||||||
.ok_or_else(|| format!("官方 URL 必须使用 https:{url}"))?;
|
|
||||||
let (host, path) = rest
|
|
||||||
.split_once('/')
|
|
||||||
.ok_or_else(|| format!("官方 URL 缺少路径:{url}"))?;
|
|
||||||
|
|
||||||
let mut relative_destination = PathBuf::from(sanitize_segment(host));
|
|
||||||
for segment in path.split('/') {
|
|
||||||
if segment.is_empty() {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if segment == "." || segment == ".." {
|
|
||||||
return Err(format!("官方 URL 包含不安全路径片段:{url}"));
|
|
||||||
}
|
|
||||||
|
|
||||||
// 官方资源 URL 不携带 query/fragment。若出现则直接拒绝,而非静默剥除——
|
|
||||||
// 否则仅 query 不同的两个 URL 会映射到同一目标文件而相互覆盖,
|
|
||||||
// 并导致每轮 hash 复用校验失配、反复重下。
|
|
||||||
if segment.contains('?') || segment.contains('#') {
|
|
||||||
return Err(format!("官方资源 URL 不允许包含 query 或 fragment:{url}"));
|
|
||||||
}
|
|
||||||
relative_destination.push(sanitize_segment(segment));
|
|
||||||
}
|
|
||||||
|
|
||||||
let destination = self.output_root.join(relative_destination);
|
|
||||||
ensure_path_within_root(&self.output_root, &destination)?;
|
ensure_path_within_root(&self.output_root, &destination)?;
|
||||||
Ok(destination)
|
Ok(destination)
|
||||||
}
|
}
|
||||||
@@ -1834,6 +2025,7 @@ fn default_download_quarantine_version() -> u32 {
|
|||||||
|
|
||||||
/// Phase A 产出的单个下载计划项:URL、目标路径,以及若命中本地 manifest
|
/// Phase A 产出的单个下载计划项:URL、目标路径,以及若命中本地 manifest
|
||||||
/// 校验则带上「已验证可跳过」的结果(`existing`)。
|
/// 校验则带上「已验证可跳过」的结果(`existing`)。
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
struct PlannedDownload {
|
struct PlannedDownload {
|
||||||
url: String,
|
url: String,
|
||||||
destination: PathBuf,
|
destination: PathBuf,
|
||||||
@@ -1854,6 +2046,21 @@ struct PullOneError {
|
|||||||
retry_error: Option<CurlRetryError>,
|
retry_error: Option<CurlRetryError>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
struct CurlDownloadBackend<'a> {
|
||||||
|
service: &'a OfficialResourcePullService,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DownloaderBackend<PlannedDownload> for CurlDownloadBackend<'_> {
|
||||||
|
type Output = PullOneResult;
|
||||||
|
type Error = PullOneError;
|
||||||
|
|
||||||
|
fn download(&self, task: PlannedDownload) -> Result<Self::Output, Self::Error> {
|
||||||
|
task.existing
|
||||||
|
.map(Ok)
|
||||||
|
.unwrap_or_else(|| self.service.pull_one(&task.url, &task.destination))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl PullOneError {
|
impl PullOneError {
|
||||||
fn plain(message: String) -> Self {
|
fn plain(message: String) -> Self {
|
||||||
Self {
|
Self {
|
||||||
@@ -2088,20 +2295,6 @@ fn read_u32_le(bytes: &[u8], offset: usize) -> u32 {
|
|||||||
])
|
])
|
||||||
}
|
}
|
||||||
|
|
||||||
fn sanitize_segment(segment: &str) -> String {
|
|
||||||
segment
|
|
||||||
.chars()
|
|
||||||
.map(|ch| {
|
|
||||||
if ch.is_control() || matches!(ch, '/' | '\\' | ':' | '*' | '?' | '"' | '<' | '>' | '|')
|
|
||||||
{
|
|
||||||
'_'
|
|
||||||
} else {
|
|
||||||
ch
|
|
||||||
}
|
|
||||||
})
|
|
||||||
.collect()
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
@@ -3263,21 +3456,26 @@ exit 22
|
|||||||
.official_hash
|
.official_hash
|
||||||
.as_ref()
|
.as_ref()
|
||||||
.expect("verification event must carry official hash detail");
|
.expect("verification event must carry official hash detail");
|
||||||
let hash_finished_index = events
|
let pair_finished_indices = [hash.data_url.as_str(), hash.hash_url.as_str()]
|
||||||
.iter()
|
.into_iter()
|
||||||
.position(|event| {
|
.map(|url| {
|
||||||
event.kind == OfficialResourcePullProgressKind::Finished
|
events
|
||||||
&& event.url == hash.hash_url
|
.iter()
|
||||||
|
.position(|event| {
|
||||||
|
event.kind == OfficialResourcePullProgressKind::Finished
|
||||||
|
&& event.url == url
|
||||||
|
})
|
||||||
|
.expect("hash pair member must finish before verification")
|
||||||
})
|
})
|
||||||
.expect("hash sidecar must finish before verification");
|
.collect::<Vec<_>>();
|
||||||
let verification_index = events
|
let verification_index = events
|
||||||
.iter()
|
.iter()
|
||||||
.position(|event| std::ptr::eq(event, *verification_event))
|
.position(|event| std::ptr::eq(event, *verification_event))
|
||||||
.expect("verification event must be present in event stream");
|
.expect("verification event must be present in event stream");
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
verification_index,
|
verification_index,
|
||||||
hash_finished_index + 1,
|
pair_finished_indices.into_iter().max().unwrap() + 1,
|
||||||
"official hash verification must run immediately after sidecar is complete"
|
"official hash verification must run immediately after the hash pair is complete"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -3320,7 +3518,8 @@ exit 22
|
|||||||
write_fake_curl(&curl_path);
|
write_fake_curl(&curl_path);
|
||||||
|
|
||||||
// 顺序下载:每个 URL 恰好一次 started + 一次 finished,全部文件落盘。
|
// 顺序下载:每个 URL 恰好一次 started + 一次 finished,全部文件落盘。
|
||||||
let service = OfficialResourcePullService::with_curl_command(out_dir.path(), &curl_path);
|
let service = OfficialResourcePullService::with_curl_command(out_dir.path(), &curl_path)
|
||||||
|
.with_max_concurrency(1);
|
||||||
let plan = build_official_pull_plan_for_platforms(
|
let plan = build_official_pull_plan_for_platforms(
|
||||||
discovery_plan(),
|
discovery_plan(),
|
||||||
inventory(),
|
inventory(),
|
||||||
@@ -3354,6 +3553,71 @@ exit 22
|
|||||||
assert_eq!(manifest.entries.len(), all_urls.len());
|
assert_eq!(manifest.entries.len(), all_urls.len());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn downloads_run_with_bounded_concurrency_and_keep_report_order() {
|
||||||
|
let out_dir = TempDir::new().unwrap();
|
||||||
|
let bin_dir = TempDir::new().unwrap();
|
||||||
|
let curl_path = bin_dir.path().join("curl");
|
||||||
|
write_fake_curl(&curl_path);
|
||||||
|
|
||||||
|
let service = OfficialResourcePullService::with_curl_command(out_dir.path(), &curl_path)
|
||||||
|
.with_max_concurrency(3);
|
||||||
|
let plan = build_official_pull_plan_for_platforms(
|
||||||
|
discovery_plan(),
|
||||||
|
inventory(),
|
||||||
|
&[PatchPlatform::Windows],
|
||||||
|
);
|
||||||
|
let all_urls = plan.all_urls().unwrap();
|
||||||
|
let mut events = Vec::new();
|
||||||
|
let report = service
|
||||||
|
.pull_with_progress(&plan, |event| events.push(event))
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
assert_eq!(service.max_concurrency(), 3);
|
||||||
|
assert_eq!(report.items.len(), all_urls.len());
|
||||||
|
assert_eq!(
|
||||||
|
report
|
||||||
|
.items
|
||||||
|
.iter()
|
||||||
|
.map(|item| &item.url)
|
||||||
|
.collect::<Vec<_>>(),
|
||||||
|
all_urls.iter().collect::<Vec<_>>()
|
||||||
|
);
|
||||||
|
for url in &all_urls {
|
||||||
|
assert_eq!(
|
||||||
|
events
|
||||||
|
.iter()
|
||||||
|
.filter(|event| {
|
||||||
|
event.url == *url && event.kind == OfficialResourcePullProgressKind::Started
|
||||||
|
})
|
||||||
|
.count(),
|
||||||
|
1,
|
||||||
|
"url {url} 的 started 次数"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
events
|
||||||
|
.iter()
|
||||||
|
.filter(|event| {
|
||||||
|
event.url == *url
|
||||||
|
&& event.kind == OfficialResourcePullProgressKind::Finished
|
||||||
|
})
|
||||||
|
.count(),
|
||||||
|
1,
|
||||||
|
"url {url} 的 finished 次数"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let finished_indices = events
|
||||||
|
.iter()
|
||||||
|
.filter(|event| event.kind == OfficialResourcePullProgressKind::Finished)
|
||||||
|
.map(|event| event.index)
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
assert_eq!(finished_indices, (1..=all_urls.len()).collect::<Vec<_>>());
|
||||||
|
assert_eq!(
|
||||||
|
service.read_download_manifest().unwrap().entries.len(),
|
||||||
|
all_urls.len()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn retries_transient_download_failures() {
|
fn retries_transient_download_failures() {
|
||||||
let out_dir = TempDir::new().unwrap();
|
let out_dir = TempDir::new().unwrap();
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ use crate::path_security::{
|
|||||||
ensure_path_within_root, ensure_safe_file_target, read_file_no_symlink, write_file_atomic,
|
ensure_path_within_root, ensure_safe_file_target, read_file_no_symlink, write_file_atomic,
|
||||||
STATE_FILE_MODE,
|
STATE_FILE_MODE,
|
||||||
};
|
};
|
||||||
|
use crate::sync_translation_task_repository_at;
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use std::collections::BTreeMap;
|
use std::collections::BTreeMap;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
@@ -121,6 +122,10 @@ pub struct OfficialTextUnitTaskQuery {
|
|||||||
pub text_unit_format: Option<String>,
|
pub text_unit_format: Option<String>,
|
||||||
/// Filter tasks by whether a diagnostic reason is present.
|
/// Filter tasks by whether a diagnostic reason is present.
|
||||||
pub has_reason: Option<bool>,
|
pub has_reason: Option<bool>,
|
||||||
|
/// Filter tasks by whether a provider failure reason is present.
|
||||||
|
pub has_failure_reason: Option<bool>,
|
||||||
|
/// Filter by mutable provider-worker status.
|
||||||
|
pub task_status: Option<String>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Aggregate counters for an incremental TextUnit task queue.
|
/// Aggregate counters for an incremental TextUnit task queue.
|
||||||
@@ -355,6 +360,16 @@ pub fn write_official_textunit_queues(
|
|||||||
write_textunit_task_queue_at(resource_root, &task_queue)?;
|
write_textunit_task_queue_at(resource_root, &task_queue)?;
|
||||||
|
|
||||||
let task_queue_path = resource_root.join(OFFICIAL_TEXTUNIT_TASK_QUEUE_FILE);
|
let task_queue_path = resource_root.join(OFFICIAL_TEXTUNIT_TASK_QUEUE_FILE);
|
||||||
|
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||||
|
.enable_all()
|
||||||
|
.build()
|
||||||
|
.map_err(|error| format!("构建官方 TextUnit 任务状态同步运行时失败:{error}"))?;
|
||||||
|
runtime
|
||||||
|
.block_on(sync_translation_task_repository_at(
|
||||||
|
resource_root,
|
||||||
|
&task_queue,
|
||||||
|
))
|
||||||
|
.map_err(|error| format!("同步官方 TextUnit 任务状态到 SQLite 失败:{error}"))?;
|
||||||
let crowdin_queue =
|
let crowdin_queue =
|
||||||
CrowdinTextUnitQueue::from_textunit_task_queue(task_queue_path.clone(), &task_queue);
|
CrowdinTextUnitQueue::from_textunit_task_queue(task_queue_path.clone(), &task_queue);
|
||||||
write_crowdin_textunit_queue_at(resource_root, &crowdin_queue)?;
|
write_crowdin_textunit_queue_at(resource_root, &crowdin_queue)?;
|
||||||
@@ -492,7 +507,10 @@ fn parse_entries_by_destination(
|
|||||||
by_destination
|
by_destination
|
||||||
}
|
}
|
||||||
|
|
||||||
fn textunit_task_matches(task: &OfficialTextUnitTask, query: &OfficialTextUnitTaskQuery) -> bool {
|
pub(crate) fn textunit_task_matches(
|
||||||
|
task: &OfficialTextUnitTask,
|
||||||
|
query: &OfficialTextUnitTaskQuery,
|
||||||
|
) -> bool {
|
||||||
if query
|
if query
|
||||||
.task_id
|
.task_id
|
||||||
.as_ref()
|
.as_ref()
|
||||||
|
|||||||
@@ -29,6 +29,9 @@ use crate::path_security::{
|
|||||||
read_file_no_symlink, validate_output_root, write_file_atomic, STATE_FILE_MODE,
|
read_file_no_symlink, validate_output_root, write_file_atomic, STATE_FILE_MODE,
|
||||||
};
|
};
|
||||||
use crate::release_flow::ReleaseFlowStatusCode;
|
use crate::release_flow::ReleaseFlowStatusCode;
|
||||||
|
use crate::translation_tasks::{
|
||||||
|
build_translation_handoff, sync_translation_task_repository_at, write_translation_handoff_at,
|
||||||
|
};
|
||||||
use crate::{
|
use crate::{
|
||||||
build_official_pull_plan_for_platform_inventory, build_official_sync_plan,
|
build_official_pull_plan_for_platform_inventory, build_official_sync_plan,
|
||||||
changed_endpoint_urls, default_official_platforms, DownloadError,
|
changed_endpoint_urls, default_official_platforms, DownloadError,
|
||||||
@@ -42,15 +45,17 @@ use crate::{
|
|||||||
read_parse_cache_at, OfficialParseCacheService, OfficialParseConfig, OfficialParseSummary,
|
read_parse_cache_at, OfficialParseCacheService, OfficialParseConfig, OfficialParseSummary,
|
||||||
};
|
};
|
||||||
use crate::{FileSystemCasRepository, SqliteResourceRepository};
|
use crate::{FileSystemCasRepository, SqliteResourceRepository};
|
||||||
|
use crate::{DEFAULT_DOWNLOAD_CONCURRENCY, MAX_DOWNLOAD_CONCURRENCY, MIN_DOWNLOAD_CONCURRENCY};
|
||||||
use bat_adapters::official::game_main_config::YostarJpGameMainConfig;
|
use bat_adapters::official::game_main_config::YostarJpGameMainConfig;
|
||||||
use bat_adapters::official::inventory::{
|
|
||||||
YostarJpPlatformCatalogInventory, YostarJpPlatformDownloadInventory,
|
|
||||||
};
|
|
||||||
use bat_adapters::official::launcher::YostarJpLauncherManifestFile;
|
use bat_adapters::official::launcher::YostarJpLauncherManifestFile;
|
||||||
use bat_adapters::official::yostar_jp::{
|
use bat_adapters::official::yostar_jp::{
|
||||||
server_info_url, PatchPlatform, YostarJpResourceDiscoveryPlan, YostarJpResourceEndpoint,
|
PatchPlatform, YostarJpResourceDiscoveryPlan, YostarJpResourceEndpoint,
|
||||||
YostarJpResourceEndpointKind, YostarJpServerInfo, YostarJpSyncSnapshot,
|
YostarJpResourceEndpointKind, YostarJpServerInfo, YostarJpSyncSnapshot,
|
||||||
};
|
};
|
||||||
|
use bat_adapters::official::{
|
||||||
|
OfficialResourceBackend, PlatformCatalogInput, YostarJpBackend,
|
||||||
|
YostarJpPlatformDownloadInventory,
|
||||||
|
};
|
||||||
use bat_core::ErrorCode;
|
use bat_core::ErrorCode;
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
@@ -115,6 +120,10 @@ pub struct OfficialUpdateConfig {
|
|||||||
pub curl_command: PathBuf,
|
pub curl_command: PathBuf,
|
||||||
/// Proxy selection used by all official `curl` transfers.
|
/// Proxy selection used by all official `curl` transfers.
|
||||||
pub curl_proxy: CurlProxyConfig,
|
pub curl_proxy: CurlProxyConfig,
|
||||||
|
/// Maximum number of resource downloads executed concurrently.
|
||||||
|
///
|
||||||
|
/// `8` is the default; values are accepted only in `1..=256`.
|
||||||
|
pub download_concurrency: usize,
|
||||||
/// Unzip command used when a metadata change requires GameMainConfig parsing.
|
/// Unzip command used when a metadata change requires GameMainConfig parsing.
|
||||||
pub unzip_command: PathBuf,
|
pub unzip_command: PathBuf,
|
||||||
/// Dry run reports decisions and optional plan URLs without writing sync state.
|
/// Dry run reports decisions and optional plan URLs without writing sync state.
|
||||||
@@ -151,6 +160,7 @@ impl Default for OfficialUpdateConfig {
|
|||||||
snapshot_path: None,
|
snapshot_path: None,
|
||||||
curl_command: PathBuf::from("curl"),
|
curl_command: PathBuf::from("curl"),
|
||||||
curl_proxy: CurlProxyConfig::default(),
|
curl_proxy: CurlProxyConfig::default(),
|
||||||
|
download_concurrency: DEFAULT_DOWNLOAD_CONCURRENCY,
|
||||||
unzip_command: PathBuf::from("unzip"),
|
unzip_command: PathBuf::from("unzip"),
|
||||||
dry_run: false,
|
dry_run: false,
|
||||||
plan: false,
|
plan: false,
|
||||||
@@ -844,6 +854,8 @@ pub struct OfficialUpdateReport {
|
|||||||
pub crowdin_textunit_queue_path: Option<PathBuf>,
|
pub crowdin_textunit_queue_path: Option<PathBuf>,
|
||||||
/// Incremental TextUnit task queue summary.
|
/// Incremental TextUnit task queue summary.
|
||||||
pub textunit_task_summary: Option<OfficialTextUnitTaskSummary>,
|
pub textunit_task_summary: Option<OfficialTextUnitTaskSummary>,
|
||||||
|
/// Versioned translation worker handoff view.
|
||||||
|
pub translation_handoff_path: Option<PathBuf>,
|
||||||
/// Snapshot path written after success.
|
/// Snapshot path written after success.
|
||||||
pub snapshot_written: Option<PathBuf>,
|
pub snapshot_written: Option<PathBuf>,
|
||||||
/// Launcher bootstrap artifact written for this run.
|
/// Launcher bootstrap artifact written for this run.
|
||||||
@@ -1221,7 +1233,8 @@ impl OfficialUpdateService {
|
|||||||
&active_resource_root,
|
&active_resource_root,
|
||||||
&config.curl_command,
|
&config.curl_command,
|
||||||
)
|
)
|
||||||
.with_proxy_config(config.curl_proxy.clone());
|
.with_proxy_config(config.curl_proxy.clone())
|
||||||
|
.with_max_concurrency(config.download_concurrency);
|
||||||
let snapshot_path = snapshot_path_for(config, &active_resource_root);
|
let snapshot_path = snapshot_path_for(config, &active_resource_root);
|
||||||
let bootstrap_cache_path = config.bootstrap_cache_path();
|
let bootstrap_cache_path = config.bootstrap_cache_path();
|
||||||
|
|
||||||
@@ -1668,6 +1681,7 @@ impl OfficialUpdateService {
|
|||||||
textunit_task_queue_path: None,
|
textunit_task_queue_path: None,
|
||||||
crowdin_textunit_queue_path: None,
|
crowdin_textunit_queue_path: None,
|
||||||
textunit_task_summary: None,
|
textunit_task_summary: None,
|
||||||
|
translation_handoff_path: None,
|
||||||
snapshot_written: None,
|
snapshot_written: None,
|
||||||
launcher_bootstrap_artifact_path: None,
|
launcher_bootstrap_artifact_path: None,
|
||||||
repository_import_enabled: config.import_repository,
|
repository_import_enabled: config.import_repository,
|
||||||
@@ -1846,7 +1860,8 @@ impl OfficialUpdateService {
|
|||||||
&publish_plan.staging_path,
|
&publish_plan.staging_path,
|
||||||
&config.curl_command,
|
&config.curl_command,
|
||||||
)
|
)
|
||||||
.with_proxy_config(config.curl_proxy.clone());
|
.with_proxy_config(config.curl_proxy.clone())
|
||||||
|
.with_max_concurrency(config.download_concurrency);
|
||||||
let pruned_stale_resource_count = staging_fetcher
|
let pruned_stale_resource_count = staging_fetcher
|
||||||
.prune_stale_manifest_entries(&pull_plan)
|
.prune_stale_manifest_entries(&pull_plan)
|
||||||
.map_err(anyhow::Error::msg)?;
|
.map_err(anyhow::Error::msg)?;
|
||||||
@@ -2201,6 +2216,7 @@ fn run_post_sync_textunit_queue_if_needed(
|
|||||||
&& is_crowdin_textunit_queue_current(resource_root, &queue)
|
&& is_crowdin_textunit_queue_current(resource_root, &queue)
|
||||||
.map_err(anyhow::Error::msg)?
|
.map_err(anyhow::Error::msg)?
|
||||||
{
|
{
|
||||||
|
sync_translation_task_repository(resource_root, &queue)?;
|
||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
"textunit",
|
"textunit",
|
||||||
format!(
|
format!(
|
||||||
@@ -2212,7 +2228,8 @@ fn run_post_sync_textunit_queue_if_needed(
|
|||||||
));
|
));
|
||||||
report.textunit_task_queue_path = Some(task_queue_path);
|
report.textunit_task_queue_path = Some(task_queue_path);
|
||||||
report.crowdin_textunit_queue_path = Some(crowdin_queue_path);
|
report.crowdin_textunit_queue_path = Some(crowdin_queue_path);
|
||||||
report.textunit_task_summary = Some(queue.summary);
|
report.textunit_task_summary = Some(queue.summary.clone());
|
||||||
|
write_translation_handoff_for_release(resource_root, &queue, report, progress)?;
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
@@ -2227,6 +2244,19 @@ fn run_post_sync_textunit_queue_if_needed(
|
|||||||
run_post_sync_textunit_queue(resource_root, report, progress, should_cancel)
|
run_post_sync_textunit_queue(resource_root, report, progress, should_cancel)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn sync_translation_task_repository(
|
||||||
|
resource_root: &Path,
|
||||||
|
queue: &crate::official_textunit_queue::OfficialTextUnitTaskQueue,
|
||||||
|
) -> anyhow::Result<()> {
|
||||||
|
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||||
|
.enable_all()
|
||||||
|
.build()?;
|
||||||
|
runtime
|
||||||
|
.block_on(sync_translation_task_repository_at(resource_root, queue))
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
fn run_post_sync_textunit_queue(
|
fn run_post_sync_textunit_queue(
|
||||||
resource_root: &Path,
|
resource_root: &Path,
|
||||||
report: &mut OfficialUpdateReport,
|
report: &mut OfficialUpdateReport,
|
||||||
@@ -2257,6 +2287,11 @@ fn run_post_sync_textunit_queue(
|
|||||||
),
|
),
|
||||||
));
|
));
|
||||||
apply_textunit_queue_report(report, queue_report);
|
apply_textunit_queue_report(report, queue_report);
|
||||||
|
if let Some(queue) =
|
||||||
|
read_textunit_task_queue_at(resource_root).map_err(anyhow::Error::msg)?
|
||||||
|
{
|
||||||
|
write_translation_handoff_for_release(resource_root, &queue, report, progress)?;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
Err(error) => {
|
Err(error) => {
|
||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
@@ -2277,6 +2312,48 @@ fn apply_textunit_queue_report(
|
|||||||
report.textunit_task_summary = Some(queue_report.summary);
|
report.textunit_task_summary = Some(queue_report.summary);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn write_translation_handoff_for_release(
|
||||||
|
resource_root: &Path,
|
||||||
|
queue: &crate::official_textunit_queue::OfficialTextUnitTaskQueue,
|
||||||
|
report: &mut OfficialUpdateReport,
|
||||||
|
progress: &mut dyn FnMut(OfficialUpdateProgress),
|
||||||
|
) -> anyhow::Result<()> {
|
||||||
|
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||||
|
.enable_all()
|
||||||
|
.build()?;
|
||||||
|
let tasks = runtime.block_on(async {
|
||||||
|
let repository = crate::translation_tasks::SqliteTranslationTaskRepository::new(
|
||||||
|
crate::translation_tasks::SqliteTranslationTaskRepository::repository_path(
|
||||||
|
resource_root,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))?;
|
||||||
|
repository
|
||||||
|
.sync_queue(queue)
|
||||||
|
.await
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))?;
|
||||||
|
repository
|
||||||
|
.list(&crate::official_textunit_queue::OfficialTextUnitTaskQuery::default())
|
||||||
|
.await
|
||||||
|
.map_err(|error| anyhow::anyhow!("{error}"))
|
||||||
|
})?;
|
||||||
|
let handoff = build_translation_handoff(queue, &tasks);
|
||||||
|
write_translation_handoff_at(resource_root, &handoff).map_err(anyhow::Error::msg)?;
|
||||||
|
let path = resource_root.join(crate::translation_tasks::TRANSLATION_HANDOFF_FILE);
|
||||||
|
report.translation_handoff_path = Some(path.clone());
|
||||||
|
progress(OfficialUpdateProgress::new(
|
||||||
|
"textunit",
|
||||||
|
format!(
|
||||||
|
"翻译 handoff 已发布:任务={} provider_run={} {}",
|
||||||
|
handoff.units.len(),
|
||||||
|
handoff.provider_runs.len(),
|
||||||
|
path.display()
|
||||||
|
),
|
||||||
|
));
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
fn run_post_sync_repository_import(
|
fn run_post_sync_repository_import(
|
||||||
config: &OfficialUpdateConfig,
|
config: &OfficialUpdateConfig,
|
||||||
resource_root: &Path,
|
resource_root: &Path,
|
||||||
@@ -2370,8 +2447,9 @@ fn build_pull_plan(
|
|||||||
should_cancel: &mut dyn FnMut() -> bool,
|
should_cancel: &mut dyn FnMut() -> bool,
|
||||||
) -> anyhow::Result<OfficialResourcePullPlan> {
|
) -> anyhow::Result<OfficialResourcePullPlan> {
|
||||||
check_shutdown_requested(should_cancel)?;
|
check_shutdown_requested(should_cancel)?;
|
||||||
let discovery = server_info
|
let backend = YostarJpBackend;
|
||||||
.discovery_plan(connection_group, app_version, platforms)
|
let discovery = backend
|
||||||
|
.discovery_plan(server_info, connection_group, app_version, platforms)
|
||||||
.map_err(anyhow::Error::msg)?;
|
.map_err(anyhow::Error::msg)?;
|
||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
"plan",
|
"plan",
|
||||||
@@ -2402,7 +2480,9 @@ fn load_server_info(
|
|||||||
match source {
|
match source {
|
||||||
OfficialServerInfoSource::LocalPath(path) => Ok(fs::read(path)?),
|
OfficialServerInfoSource::LocalPath(path) => Ok(fs::read(path)?),
|
||||||
OfficialServerInfoSource::OfficialFile(file_name) => {
|
OfficialServerInfoSource::OfficialFile(file_name) => {
|
||||||
let url = server_info_url(file_name).map_err(anyhow::Error::msg)?;
|
let url = YostarJpBackend
|
||||||
|
.server_info_url(file_name)
|
||||||
|
.map_err(anyhow::Error::msg)?;
|
||||||
fetcher.fetch_bytes(&url).map_err(anyhow::Error::new)
|
fetcher.fetch_bytes(&url).map_err(anyhow::Error::new)
|
||||||
}
|
}
|
||||||
OfficialServerInfoSource::OfficialUrl(url) => {
|
OfficialServerInfoSource::OfficialUrl(url) => {
|
||||||
@@ -2489,6 +2569,7 @@ fn waiting_for_official_resources_report(
|
|||||||
textunit_task_queue_path: None,
|
textunit_task_queue_path: None,
|
||||||
crowdin_textunit_queue_path: None,
|
crowdin_textunit_queue_path: None,
|
||||||
textunit_task_summary: None,
|
textunit_task_summary: None,
|
||||||
|
translation_handoff_path: None,
|
||||||
snapshot_written: None,
|
snapshot_written: None,
|
||||||
launcher_bootstrap_artifact_path: None,
|
launcher_bootstrap_artifact_path: None,
|
||||||
repository_import_enabled: config.import_repository,
|
repository_import_enabled: config.import_repository,
|
||||||
@@ -2894,6 +2975,12 @@ pub fn diff_extended_snapshot(
|
|||||||
}
|
}
|
||||||
|
|
||||||
fn validate_update_paths(config: &OfficialUpdateConfig) -> Result<(), String> {
|
fn validate_update_paths(config: &OfficialUpdateConfig) -> Result<(), String> {
|
||||||
|
if !(MIN_DOWNLOAD_CONCURRENCY..=MAX_DOWNLOAD_CONCURRENCY).contains(&config.download_concurrency)
|
||||||
|
{
|
||||||
|
return Err(format!(
|
||||||
|
"下载并发数必须在 {MIN_DOWNLOAD_CONCURRENCY}..={MAX_DOWNLOAD_CONCURRENCY} 范围内"
|
||||||
|
));
|
||||||
|
}
|
||||||
validate_output_root(&config.output_root)?;
|
validate_output_root(&config.output_root)?;
|
||||||
validate_output_root(&config.localized_output_root)?;
|
validate_output_root(&config.localized_output_root)?;
|
||||||
validate_separate_output_roots(&config.output_root, &config.localized_output_root)?;
|
validate_separate_output_roots(&config.output_root, &config.localized_output_root)?;
|
||||||
@@ -4047,17 +4134,14 @@ fn build_inventory_from_seed_catalogs(
|
|||||||
anyhow::anyhow!("官方发现结果缺少 {} MediaCatalog.bytes", platform.as_str())
|
anyhow::anyhow!("官方发现结果缺少 {} MediaCatalog.bytes", platform.as_str())
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
platform_catalogs.push(YostarJpPlatformCatalogInventory::from_catalog_bytes(
|
platform_catalogs.push(PlatformCatalogInput {
|
||||||
*platform,
|
platform: *platform,
|
||||||
bundle_packing_info,
|
bundle_packing_info,
|
||||||
media_catalog,
|
media_catalog,
|
||||||
));
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(YostarJpPlatformDownloadInventory::from_catalog_bytes(
|
Ok(YostarJpBackend.inventory(table_catalog, &platform_catalogs))
|
||||||
table_catalog,
|
|
||||||
platform_catalogs,
|
|
||||||
))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Debug, Default)]
|
#[derive(Debug, Default)]
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user