mirror of
https://github.com/Yuyi-Oak/BlueArchiveToolkit.git
synced 2026-07-22 01:46:44 +08:00
feat: harden official sync scheduling and integrity
This commit is contained in:
+5
-4
@@ -1,9 +1,9 @@
|
|||||||
# BlueArchiveToolkit 当前工作区状态
|
# BlueArchiveToolkit 当前工作区状态
|
||||||
|
|
||||||
- **更新时间**:2026-07-06
|
- **更新时间**:2026-07-11
|
||||||
- **状态来源**:本地工作区盘点、代码验证和最新提交
|
- **状态来源**:本地工作区盘点、代码验证和最新提交
|
||||||
- **状态分支**:`experiment`
|
- **状态分支**:`experiment`
|
||||||
- **最新功能提交**:`789402c feat: add official resource sync pipeline`
|
- **最新已推送功能提交**:`2bdc55a feat: support cancellable official sync`
|
||||||
- **权威计划**:`PROJECT_PLAN.md`
|
- **权威计划**:`PROJECT_PLAN.md`
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -21,7 +21,8 @@ Rust 侧官方日服资源链路已经从实验验证推进到正式入口:
|
|||||||
5. 下载后使用本地 manifest 的 size + BLAKE3 校验复用文件;官方 seed `.hash` 使用 `xxHash32(seed=0)` 强校验。
|
5. 下载后使用本地 manifest 的 size + BLAKE3 校验复用文件;官方 seed `.hash` 使用 `xxHash32(seed=0)` 强校验。
|
||||||
6. 支持 `.part` 断点续传、失败后 clean retry、本地 manifest audit/repair。
|
6. 支持 `.part` 断点续传、失败后 clean retry、本地 manifest audit/repair。
|
||||||
7. `bat-official-sync --watch` 可常驻运行,正常检查默认每 1 小时一次;远端和本地一致时默认静默,失败后默认 60 秒快速重试;CLI 默认向 stderr 输出 ASCII banner 和人类可读 progress log。
|
7. `bat-official-sync --watch` 可常驻运行,正常检查默认每 1 小时一次;远端和本地一致时默认静默,失败后默认 60 秒快速重试;CLI 默认向 stderr 输出 ASCII banner 和人类可读 progress log。
|
||||||
8. 非 dry-run 使用 `--output/.official-sync.lock` 防止并发写同一状态目录。
|
8. 远端 snapshot 未变化但输出目录为空时,会按首次运行执行全量拉取;官方 seed `.hash` 校验失败时会清理对应 manifest 条目,避免失败产物被后续本地 audit 误判为可复用。
|
||||||
|
9. 非 dry-run 使用 `--output/.official-sync.lock` 防止并发写同一状态目录。
|
||||||
|
|
||||||
仍需明确:这不是完整产品完成。Go CLI 最小入口、完整 AssetBundle 解析、Patch、翻译系统、API Server 和 Web 仍是后续工作;真实官方网络全量下载 smoke test 尚未记录在仓库文档中。
|
仍需明确:这不是完整产品完成。Go CLI 最小入口、完整 AssetBundle 解析、Patch、翻译系统、API Server 和 Web 仍是后续工作;真实官方网络全量下载 smoke test 尚未记录在仓库文档中。
|
||||||
|
|
||||||
@@ -241,7 +242,7 @@ cargo run -p bat-infrastructure --bin bat-official-sync -- \
|
|||||||
2. 不要指向已有游戏客户端目录。
|
2. 不要指向已有游戏客户端目录。
|
||||||
3. 不要指向 `/home/wanye/D/BlueArchive` 这类开发或人工维护资源目录。
|
3. 不要指向 `/home/wanye/D/BlueArchive` 这类开发或人工维护资源目录。
|
||||||
4. `--auto-discover` 可以下载官方 metadata,并按官方 manifest 临时获取 `resources.assets` 以解析 `GameMainConfig`;旧 ZIP manifest 才会下载临时 game zip。该流程不会安装或启动官方 launcher。
|
4. `--auto-discover` 可以下载官方 metadata,并按官方 manifest 临时获取 `resources.assets` 以解析 `GameMainConfig`;旧 ZIP manifest 才会下载临时 game zip。该流程不会安装或启动官方 launcher。
|
||||||
5. systemd service、容器或 Go 进程可以负责守护 `bat-official-sync --watch`,但定时检查逻辑已经在 Rust 内部;正常检查默认 1 小时,失败重试默认 60 秒。
|
5. systemd service、容器或 Go 进程可以负责守护 `bat-official-sync --watch`,但定时检查逻辑已经在 Rust 内部;正常检查默认 1 小时,失败重试默认 60 秒,每天北京时间(UTC+8)`03:00`、`16:00`、`18:00` 会强制执行一次自动刷新。
|
||||||
|
|
||||||
详细运行说明见 `docs/guides/official-resource-test-pull.md`。
|
详细运行说明见 `docs/guides/official-resource-test-pull.md`。
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -90,7 +90,7 @@
|
|||||||
- Unity/Manifest/Client 适配器框架。
|
- Unity/Manifest/Client 适配器框架。
|
||||||
- CAS V1:原子写入、BLAKE3 校验、引用计数、GC、并发测试和损坏检测。
|
- CAS V1:原子写入、BLAKE3 校验、引用计数、GC、并发测试和损坏检测。
|
||||||
- 文档整理和路线图重制。
|
- 文档整理和路线图重制。
|
||||||
- Rust 官方资源同步闭环:`bat-official-sync`、`--auto-discover`、`--watch`、snapshot、manifest audit/repair、官方 seed `.hash` 校验。
|
- Rust 官方资源同步闭环:`bat-official-sync`、`--auto-discover`、`--watch`、北京时间固定强制刷新、snapshot、manifest audit/repair、官方 seed `.hash` 校验。
|
||||||
- Addressables 当前真实形态 fixture/golden 覆盖。
|
- Addressables 当前真实形态 fixture/golden 覆盖。
|
||||||
- SQLite Resource Repository 和粗粒度 FFI JSON 接口。
|
- SQLite Resource Repository 和粗粒度 FFI JSON 接口。
|
||||||
|
|
||||||
|
|||||||
@@ -71,7 +71,7 @@ cargo run -p bat-infrastructure --bin bat-official-sync -- \
|
|||||||
--dry-run
|
--dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
常驻自动更新,正常情况下默认每 1 小时检查一次;远端和本地一致时静默。下载或发现失败时默认 60 秒后重试,可用 `--error-retry 60s` 调整:
|
常驻自动更新,正常情况下默认每 1 小时检查一次;每天北京时间(UTC+8)`03:00`、`16:00`、`18:00` 会强制执行一次自动刷新。远端和本地一致时静默。下载或发现失败时默认 60 秒后重试,可用 `--error-retry 60s` 调整:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cargo run -p bat-infrastructure --bin bat-official-sync -- \
|
cargo run -p bat-infrastructure --bin bat-official-sync -- \
|
||||||
|
|||||||
@@ -116,7 +116,7 @@ official-sync-snapshot.json + official-download-manifest.json
|
|||||||
- 不安装、不执行官方 launcher。
|
- 不安装、不执行官方 launcher。
|
||||||
- 默认平台 `Windows + Android`。
|
- 默认平台 `Windows + Android`。
|
||||||
- `--auto-discover` 自动获取 `app-version`、`connection-group` 和 `server-info`。
|
- `--auto-discover` 自动获取 `app-version`、`connection-group` 和 `server-info`。
|
||||||
- `--watch` 常驻检查,正常检查默认 1 小时,失败重试默认 60 秒。
|
- `--watch` 常驻检查,正常检查默认 1 小时,失败重试默认 60 秒,每天北京时间(UTC+8)`03:00`、`16:00`、`18:00` 强制刷新一次。
|
||||||
- 远端 marker 无变化且本地 manifest clean 时不下载。
|
- 远端 marker 无变化且本地 manifest clean 时不下载。
|
||||||
- 本地文件损坏时 repair。
|
- 本地文件损坏时 repair。
|
||||||
- 官方 seed `.hash` 强校验;Addressables `catalog_*.hash` 作为变更 marker。
|
- 官方 seed `.hash` 强校验;Addressables `catalog_*.hash` 作为变更 marker。
|
||||||
|
|||||||
@@ -126,11 +126,12 @@
|
|||||||
5. 缺少清单、清单不匹配或文件损坏时重新下载。
|
5. 缺少清单、清单不匹配或文件损坏时重新下载。
|
||||||
6. `TableCatalog.bytes`、`BundlePackingInfo.bytes`、`MediaCatalog.bytes` 总是刷新并用官方 `.hash` 强校验;该 `.hash` 是 `xxHash32(seed=0)` 的十进制文本。
|
6. `TableCatalog.bytes`、`BundlePackingInfo.bytes`、`MediaCatalog.bytes` 总是刷新并用官方 `.hash` 强校验;该 `.hash` 是 `xxHash32(seed=0)` 的十进制文本。
|
||||||
7. `catalog_*.hash` 当前只作为 Addressables catalog 变更标记,不作为 zip/JSON 内容校验算法;Unity Addressables/SBP builder 对 JSON/bin catalog 使用 `HashingMethods.Calculate` 生成 `Hash128` 文本,运行时用它判断 remote catalog cache 是否过期,它不能套用 seed catalog 的 `xxHash32` 规则。
|
7. `catalog_*.hash` 当前只作为 Addressables catalog 变更标记,不作为 zip/JSON 内容校验算法;Unity Addressables/SBP builder 对 JSON/bin catalog 使用 `HashingMethods.Calculate` 生成 `Hash128` 文本,运行时用它判断 remote catalog cache 是否过期,它不能套用 seed catalog 的 `xxHash32` 规则。
|
||||||
8. 存在 `.part` 临时文件时通过 `curl --continue-at -` 尝试断点续传。
|
8. 官方 seed `.hash` 校验失败会让当前下载失败,并移除对应 data/hash URL 的本地 manifest 条目,避免失败产物在下一轮被本地 BLAKE3 audit 误判为健康缓存。
|
||||||
9. 新下载写入 `.part`,成功后原子 rename 到最终路径。
|
9. 存在 `.part` 临时文件时通过 `curl --continue-at -` 尝试断点续传。
|
||||||
10. 成功下载后更新本地下载清单。
|
10. 新下载写入 `.part`,成功后原子 rename 到最终路径。
|
||||||
11. 记录最终文件大小、本次传输字节数、官方 hash 校验数和执行状态。
|
11. 成功下载后更新本地下载清单。
|
||||||
12. 非官方 URL 直接拒绝。
|
12. 记录最终文件大小、本次传输字节数、官方 hash 校验数和执行状态。
|
||||||
|
13. 非官方 URL 直接拒绝。
|
||||||
|
|
||||||
路径映射时会做分段清理,避免把不安全路径写进输出目录。
|
路径映射时会做分段清理,避免把不安全路径写进输出目录。
|
||||||
|
|
||||||
@@ -186,11 +187,13 @@
|
|||||||
3. 生成当前 v2 snapshot,记录 `app_version`、`connection_group`、`bundle_version`、`addressables_root`、endpoint URL、seed `.hash` 内容、`catalog_*.hash` marker、launcher metadata 摘要和 `GameMainConfig` 摘要。
|
3. 生成当前 v2 snapshot,记录 `app_version`、`connection_group`、`bundle_version`、`addressables_root`、endpoint URL、seed `.hash` 内容、`catalog_*.hash` marker、launcher metadata 摘要和 `GameMainConfig` 摘要。
|
||||||
4. 读取上一次成功同步写出的 snapshot。
|
4. 读取上一次成功同步写出的 snapshot。
|
||||||
5. 使用 `OfficialSyncPlan` 和扩展 snapshot diff 判断是否需要下载;URL 未变但 `.hash` / marker 内容变化也会触发更新。
|
5. 使用 `OfficialSyncPlan` 和扩展 snapshot diff 判断是否需要下载;URL 未变但 `.hash` / marker 内容变化也会触发更新。
|
||||||
6. 远端无变化时执行本地 download manifest audit,检查路径、size 和 BLAKE3。
|
6. 每轮都会基于最新 seed catalog 构建当前 pull plan,并检查输出目录是否已有当前 plan 的 manifest 条目或目标文件。
|
||||||
7. 远端变化或本地 audit 发现 repair_needed 时生成 pull plan,下载并校验官方 URL。
|
7. 如果远端 snapshot 未变化但输出目录没有任何当前 plan 的本地资源,仍按首次运行处理并执行全量拉取。
|
||||||
8. 下载成功后写回新的 snapshot。
|
8. 远端无变化且本地已有资源时执行 download manifest audit,检查路径、size 和 BLAKE3。
|
||||||
|
9. 远端变化、本地 audit 发现 repair_needed,或首次空目录运行时,下载并校验官方 URL。
|
||||||
|
10. 下载成功后写回新的 snapshot。
|
||||||
|
|
||||||
该入口不安装、不执行官方启动器,也不读取生产外的本地客户端目录。Rust 正式 binary 支持 `--watch` 常驻模式,正常情况下默认每 1 小时执行一次检查;远端和本地一致时静默等待下次检查,不一致时自动下载或 repair。下载、发现或校验失败时不等待完整正常周期,默认 60 秒后重试,可用 `--error-retry` 或 `--error-retry-seconds` 调整。单次运行仍保留为核心幂等路径,systemd service、容器或 Go 进程可以只负责守护该常驻进程;cron/systemd timer 调单次模式只是可选集成方式。项目是否热更新、热重载或重启进程,由上层业务集成决定。
|
该入口不安装、不执行官方启动器,也不读取生产外的本地客户端目录。Rust 正式 binary 支持 `--watch` 常驻模式,正常情况下默认每 1 小时执行一次检查;每天北京时间(UTC+8)`03:00`、`16:00`、`18:00` 会中断普通 sleep 并强制执行一次自动刷新,该轮注入 `force=true`。远端和本地一致时静默等待下次检查,不一致时自动下载或 repair。下载、发现或校验失败时不等待完整正常周期,默认 60 秒后重试;如果固定时间强制刷新失败,会保留 pending force 并按失败重试周期继续重试,可用 `--error-retry` 或 `--error-retry-seconds` 调整。单次运行仍保留为核心幂等路径,systemd service、容器或 Go 进程可以只负责守护该常驻进程;cron/systemd timer 调单次模式只是可选集成方式。项目是否热更新、热重载或重启进程,由上层业务集成决定。
|
||||||
|
|
||||||
对应实现主要在:
|
对应实现主要在:
|
||||||
|
|
||||||
|
|||||||
@@ -157,7 +157,7 @@ target/release/bat-official-sync
|
|||||||
--watch
|
--watch
|
||||||
```
|
```
|
||||||
|
|
||||||
`--watch` 是 Rust 内部持久检查模式,正常情况下默认每 1 小时执行一次检查。远端和本地一致时默认静默;有远端变化或本地文件损坏时自动下载或 repair,并输出 JSON report。下载、发现或校验失败时默认 60 秒后重试,可显式加 `--error-retry 60s` 或 `--error-retry-seconds 60` 调整。默认 ASCII banner 和进度日志写到 stderr,成功 JSON report 写到 stdout;如果由上层服务严格解析 stderr/stdout,可加 `--no-progress`,只想关闭横幅可加 `--no-banner`。
|
`--watch` 是 Rust 内部持久检查模式,正常情况下默认每 1 小时执行一次检查,并且每天北京时间(UTC+8)`03:00`、`16:00`、`18:00` 会强制执行一次自动刷新。固定时间刷新会中断普通 interval 的 sleep,该轮注入 `force=true`;如果失败,会按失败重试周期继续重试。远端和本地一致时默认静默;有远端变化或本地文件损坏时自动下载或 repair,并输出 JSON report。下载、发现或校验失败时默认 60 秒后重试,可显式加 `--error-retry 60s` 或 `--error-retry-seconds 60` 调整。默认 ASCII banner 和进度日志写到 stderr,成功 JSON report 写到 stdout;如果由上层服务严格解析 stderr/stdout,可加 `--no-progress`,只想关闭横幅可加 `--no-banner`。
|
||||||
|
|
||||||
### systemd service 示例
|
### systemd service 示例
|
||||||
|
|
||||||
|
|||||||
@@ -157,8 +157,10 @@ cargo run -p bat-infrastructure --example official_pull_plan -- \
|
|||||||
|
|
||||||
- 每轮都会先拉取轻量官方 metadata 和 `.hash` marker。
|
- 每轮都会先拉取轻量官方 metadata 和 `.hash` marker。
|
||||||
- URL 没变但 `.hash` / marker 内容变更时,也会判定为需要更新。
|
- URL 没变但 `.hash` / marker 内容变更时,也会判定为需要更新。
|
||||||
|
- 即使远端 snapshot 未变化,只要输出目录没有任何当前 pull plan 的本地 manifest 条目或目标文件,也会按首次运行处理并执行全量拉取。
|
||||||
- 远端无变化时,默认执行本地 `official-download-manifest.json` audit。
|
- 远端无变化时,默认执行本地 `official-download-manifest.json` audit。
|
||||||
- 本地文件缺失、路径不一致、size 不一致或 BLAKE3 不一致时,默认进入 repair 并重下必要文件。
|
- 本地文件缺失、路径不一致、size 不一致或 BLAKE3 不一致时,默认进入 repair 并重下必要文件。
|
||||||
|
- 官方 seed `.hash` 校验失败会让本轮失败,并清理对应本地 manifest 条目;下一轮会继续把这类文件视为需要 repair,而不是把失败产物当作健康缓存复用。
|
||||||
- 远端和本地都一致:单次模式输出 `update_status=up_to_date`,watch 模式默认静默并等待下次检查。
|
- 远端和本地都一致:单次模式输出 `update_status=up_to_date`,watch 模式默认静默并等待下次检查。
|
||||||
- 有远端变化或本地 repair:生成 pull plan,下载完整官方资源,成功后更新 snapshot。
|
- 有远端变化或本地 repair:生成 pull plan,下载完整官方资源,成功后更新 snapshot。
|
||||||
- `--dry-run`:只报告本次是否会下载,不写 snapshot;如果 cache miss,也不会写入新的 bootstrap cache。
|
- `--dry-run`:只报告本次是否会下载,不写 snapshot;如果 cache miss,也不会写入新的 bootstrap cache。
|
||||||
@@ -229,7 +231,7 @@ cargo run -p bat-infrastructure --bin bat-official-sync -- \
|
|||||||
--error-retry 60s
|
--error-retry 60s
|
||||||
```
|
```
|
||||||
|
|
||||||
`--interval` 是正常检查周期,默认 `1h`;`--error-retry` 是下载、发现或校验失败后的重试周期,默认 `60s`,也可以用 `--error-retry-seconds 60`。CLI 默认启动时向 stderr 打印 `BlueArchiveToolkit` ASCII banner,并把人类可读的阶段进度日志写到 stderr,包括自动发现、server-info、marker、catalog、audit、download 和 snapshot 阶段;成功时 stdout 输出稳定 JSON report。需要纯机器输出时加 `--no-progress`,需要显式开启则用 `--progress`;只想关闭横幅但保留日志时可加 `--no-banner`。错误时 stderr 输出 JSON error,watch 模式下错误 JSON 的 `next_retry_seconds` 使用失败重试周期;如果未关闭 progress,错误 JSON 前可能已有 banner 和进度日志。普通错误 exit `1`,状态目录锁冲突 exit `75`。
|
`--interval` 是正常检查周期,默认 `1h`;watch 模式还会在每天北京时间(UTC+8)`03:00`、`16:00`、`18:00` 强制执行一次自动刷新,该轮会注入 `force=true`,并且会中断普通 interval 的 sleep。`--error-retry` 是下载、发现或校验失败后的重试周期,默认 `60s`,也可以用 `--error-retry-seconds 60`。CLI 默认启动时向 stderr 打印 `BlueArchiveToolkit` ASCII banner,并把人类可读的阶段进度日志写到 stderr,包括自动发现、server-info、marker、catalog、audit、download 和 snapshot 阶段;成功时 stdout 输出稳定 JSON report。需要纯机器输出时加 `--no-progress`,需要显式开启则用 `--progress`;只想关闭横幅但保留日志时可加 `--no-banner`。错误时 stderr 输出 JSON error,watch 模式下错误 JSON 的 `next_retry_seconds` 使用失败重试周期;如果未关闭 progress,错误 JSON 前可能已有 banner 和进度日志。普通错误 exit `1`,状态目录锁冲突 exit `75`。
|
||||||
|
|
||||||
生产可以直接运行 `--watch`,也可以用 systemd service、容器或 Go 进程守护它。cron/systemd timer 仍可调用单次模式,但不再是 Rust 自动更新的唯一方式。项目是否热更新、热重载或重启进程,由上层业务集成决定。生产目录应使用独立输出目录,不要指向现有客户端或人工维护的资源目录。非 dry-run 每轮会创建 `--output/.official-sync.lock`,防止并发写同一状态目录。
|
生产可以直接运行 `--watch`,也可以用 systemd service、容器或 Go 进程守护它。cron/systemd timer 仍可调用单次模式,但不再是 Rust 自动更新的唯一方式。项目是否热更新、热重载或重启进程,由上层业务集成决定。生产目录应使用独立输出目录,不要指向现有客户端或人工维护的资源目录。非 dry-run 每轮会创建 `--output/.official-sync.lock`,防止并发写同一状态目录。
|
||||||
|
|
||||||
|
|||||||
@@ -7,12 +7,16 @@ use serde::Serialize;
|
|||||||
use std::env;
|
use std::env;
|
||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
use std::thread;
|
use std::thread;
|
||||||
use std::time::{Duration, Instant};
|
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
||||||
|
|
||||||
const EXIT_ERROR: i32 = 1;
|
const EXIT_ERROR: i32 = 1;
|
||||||
const EXIT_LOCKED: i32 = 75;
|
const EXIT_LOCKED: i32 = 75;
|
||||||
const DEFAULT_WATCH_INTERVAL_SECONDS: u64 = 60 * 60;
|
const DEFAULT_WATCH_INTERVAL_SECONDS: u64 = 60 * 60;
|
||||||
const DEFAULT_ERROR_RETRY_SECONDS: u64 = 60;
|
const DEFAULT_ERROR_RETRY_SECONDS: u64 = 60;
|
||||||
|
const SECONDS_PER_DAY: u64 = 24 * 60 * 60;
|
||||||
|
const BEIJING_UTC_OFFSET_SECONDS: u64 = 8 * 60 * 60;
|
||||||
|
const DAILY_FORCED_REFRESH_LOCAL_SECONDS: [u64; 3] = [3 * 60 * 60, 16 * 60 * 60, 18 * 60 * 60];
|
||||||
|
const DAILY_FORCED_REFRESH_LABEL: &str = "UTC+8 03:00, 16:00, 18:00";
|
||||||
const STARTUP_BANNER: &str = r#"
|
const STARTUP_BANNER: &str = r#"
|
||||||
============================================================
|
============================================================
|
||||||
____ _ _ _ _ _____ _ _ _ _
|
____ _ _ _ _ _____ _ _ _ _
|
||||||
@@ -120,25 +124,64 @@ fn run_watch(options: CliOptions) -> anyhow::Result<()> {
|
|||||||
}
|
}
|
||||||
let service = OfficialUpdateService::new();
|
let service = OfficialUpdateService::new();
|
||||||
let mut logger = ProgressLogger::new(options.progress);
|
let mut logger = ProgressLogger::new(options.progress);
|
||||||
|
let mut next_forced_refresh_at = next_forced_refresh_at_or_after(SystemTime::now());
|
||||||
|
let mut pending_scheduled_force = false;
|
||||||
|
logger.log_text(
|
||||||
|
"watch",
|
||||||
|
format!(
|
||||||
|
"daily forced refresh schedule: {DAILY_FORCED_REFRESH_LABEL}; next forced refresh in {}",
|
||||||
|
format_duration(duration_until(next_forced_refresh_at, SystemTime::now()))
|
||||||
|
),
|
||||||
|
);
|
||||||
loop {
|
loop {
|
||||||
|
let now = SystemTime::now();
|
||||||
|
if now >= next_forced_refresh_at {
|
||||||
|
pending_scheduled_force = true;
|
||||||
|
logger.log_text(
|
||||||
|
"watch",
|
||||||
|
format!("scheduled forced refresh triggered at {DAILY_FORCED_REFRESH_LABEL}"),
|
||||||
|
);
|
||||||
|
next_forced_refresh_at = next_forced_refresh_after(now);
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut iteration_config = options.config.clone();
|
||||||
|
if pending_scheduled_force {
|
||||||
|
iteration_config.force = true;
|
||||||
|
}
|
||||||
|
|
||||||
let mut sleep_for = options.interval;
|
let mut sleep_for = options.interval;
|
||||||
logger.log_text("watch", "starting watch iteration");
|
logger.log_text(
|
||||||
match service.run_with_progress(&options.config, |event| logger.log(event)) {
|
"watch",
|
||||||
|
if pending_scheduled_force {
|
||||||
|
"starting watch iteration with scheduled force refresh"
|
||||||
|
} else {
|
||||||
|
"starting watch iteration"
|
||||||
|
},
|
||||||
|
);
|
||||||
|
match service.run_with_progress(&iteration_config, |event| logger.log(event)) {
|
||||||
Ok(report) => {
|
Ok(report) => {
|
||||||
|
if pending_scheduled_force {
|
||||||
|
pending_scheduled_force = false;
|
||||||
|
}
|
||||||
if should_print_status(report.update_status, options.quiet_up_to_date) {
|
if should_print_status(report.update_status, options.quiet_up_to_date) {
|
||||||
println!("{}", serde_json::to_string_pretty(&report)?);
|
println!("{}", serde_json::to_string_pretty(&report)?);
|
||||||
}
|
}
|
||||||
|
sleep_for =
|
||||||
|
sleep_for.min(duration_until(next_forced_refresh_at, SystemTime::now()));
|
||||||
logger.log_text(
|
logger.log_text(
|
||||||
"watch",
|
"watch",
|
||||||
format!(
|
format!(
|
||||||
"iteration finished with status={}; next check in {}",
|
"iteration finished with status={}; next check in {}; next scheduled force refresh in {}",
|
||||||
report.update_status.as_str(),
|
report.update_status.as_str(),
|
||||||
format_duration(sleep_for)
|
format_duration(sleep_for),
|
||||||
|
format_duration(duration_until(next_forced_refresh_at, SystemTime::now()))
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
Err(error) => {
|
Err(error) => {
|
||||||
sleep_for = options.error_retry_interval;
|
sleep_for = options.error_retry_interval;
|
||||||
|
sleep_for =
|
||||||
|
sleep_for.min(duration_until(next_forced_refresh_at, SystemTime::now()));
|
||||||
logger.log_text(
|
logger.log_text(
|
||||||
"watch",
|
"watch",
|
||||||
format!(
|
format!(
|
||||||
@@ -161,6 +204,62 @@ fn run_watch(options: CliOptions) -> anyhow::Result<()> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn next_forced_refresh_at_or_after(now: SystemTime) -> SystemTime {
|
||||||
|
forced_refresh_time(now, ForcedRefreshBoundary::AtOrAfter)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn next_forced_refresh_after(now: SystemTime) -> SystemTime {
|
||||||
|
forced_refresh_time(now, ForcedRefreshBoundary::After)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
enum ForcedRefreshBoundary {
|
||||||
|
AtOrAfter,
|
||||||
|
After,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn forced_refresh_time(now: SystemTime, boundary: ForcedRefreshBoundary) -> SystemTime {
|
||||||
|
let local_seconds = beijing_local_seconds_since_epoch(now);
|
||||||
|
let local_day = local_seconds / SECONDS_PER_DAY;
|
||||||
|
let local_second_of_day = local_seconds % SECONDS_PER_DAY;
|
||||||
|
|
||||||
|
for scheduled_second in DAILY_FORCED_REFRESH_LOCAL_SECONDS {
|
||||||
|
let matches_boundary = match boundary {
|
||||||
|
ForcedRefreshBoundary::AtOrAfter => scheduled_second >= local_second_of_day,
|
||||||
|
ForcedRefreshBoundary::After => scheduled_second > local_second_of_day,
|
||||||
|
};
|
||||||
|
if matches_boundary {
|
||||||
|
return system_time_from_beijing_local_seconds(
|
||||||
|
local_day
|
||||||
|
.saturating_mul(SECONDS_PER_DAY)
|
||||||
|
.saturating_add(scheduled_second),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
system_time_from_beijing_local_seconds(
|
||||||
|
local_day
|
||||||
|
.saturating_add(1)
|
||||||
|
.saturating_mul(SECONDS_PER_DAY)
|
||||||
|
.saturating_add(DAILY_FORCED_REFRESH_LOCAL_SECONDS[0]),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn beijing_local_seconds_since_epoch(time: SystemTime) -> u64 {
|
||||||
|
time.duration_since(UNIX_EPOCH)
|
||||||
|
.unwrap_or_default()
|
||||||
|
.as_secs()
|
||||||
|
.saturating_add(BEIJING_UTC_OFFSET_SECONDS)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn system_time_from_beijing_local_seconds(local_seconds: u64) -> SystemTime {
|
||||||
|
UNIX_EPOCH + Duration::from_secs(local_seconds.saturating_sub(BEIJING_UTC_OFFSET_SECONDS))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn duration_until(deadline: SystemTime, now: SystemTime) -> Duration {
|
||||||
|
deadline.duration_since(now).unwrap_or_default()
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
struct ProgressLogger {
|
struct ProgressLogger {
|
||||||
enabled: bool,
|
enabled: bool,
|
||||||
@@ -365,6 +464,9 @@ fn print_usage(binary: &str) {
|
|||||||
[--curl curl] [--unzip unzip] [--dry-run] [--plan] [--force] [--audit-local|--no-audit-local] [--repair|--no-repair] \
|
[--curl curl] [--unzip unzip] [--dry-run] [--plan] [--force] [--audit-local|--no-audit-local] [--repair|--no-repair] \
|
||||||
[--watch] [--interval 1h|60m|3600s] [--error-retry 60s] [--quiet-up-to-date|--no-quiet-up-to-date] [--progress|--no-progress] [--banner|--no-banner]"
|
[--watch] [--interval 1h|60m|3600s] [--error-retry 60s] [--quiet-up-to-date|--no-quiet-up-to-date] [--progress|--no-progress] [--banner|--no-banner]"
|
||||||
);
|
);
|
||||||
|
eprintln!(
|
||||||
|
"watch mode also runs scheduled force refreshes every day at {DAILY_FORCED_REFRESH_LABEL}"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
fn parse_duration(value: &str) -> anyhow::Result<Duration> {
|
fn parse_duration(value: &str) -> anyhow::Result<Duration> {
|
||||||
@@ -616,6 +718,35 @@ mod tests {
|
|||||||
assert_eq!(parse_duration("3600").unwrap(), Duration::from_secs(3600));
|
assert_eq!(parse_duration("3600").unwrap(), Duration::from_secs(3600));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn scheduled_forced_refresh_uses_beijing_local_times() {
|
||||||
|
let day = 20_000;
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
next_forced_refresh_at_or_after(beijing_time(day, 2, 30, 0)),
|
||||||
|
beijing_time(day, 3, 0, 0)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
next_forced_refresh_at_or_after(beijing_time(day, 3, 0, 0)),
|
||||||
|
beijing_time(day, 3, 0, 0)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
next_forced_refresh_after(beijing_time(day, 3, 0, 0)),
|
||||||
|
beijing_time(day, 16, 0, 0)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
next_forced_refresh_after(beijing_time(day, 18, 0, 0)),
|
||||||
|
beijing_time(day + 1, 3, 0, 0)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
duration_until(
|
||||||
|
next_forced_refresh_at_or_after(beijing_time(day, 17, 45, 0)),
|
||||||
|
beijing_time(day, 17, 45, 0)
|
||||||
|
),
|
||||||
|
Duration::from_secs(15 * 60)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn formats_progress_durations() {
|
fn formats_progress_durations() {
|
||||||
assert_eq!(format_duration(Duration::from_millis(7)), "00:00.007");
|
assert_eq!(format_duration(Duration::from_millis(7)), "00:00.007");
|
||||||
@@ -642,4 +773,13 @@ mod tests {
|
|||||||
);
|
);
|
||||||
assert_eq!(OfficialUpdateStatus::Downloaded.as_str(), "downloaded");
|
assert_eq!(OfficialUpdateStatus::Downloaded.as_str(), "downloaded");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn beijing_time(day: u64, hour: u64, minute: u64, second: u64) -> SystemTime {
|
||||||
|
system_time_from_beijing_local_seconds(
|
||||||
|
day.saturating_mul(SECONDS_PER_DAY)
|
||||||
|
.saturating_add(hour.saturating_mul(60 * 60))
|
||||||
|
.saturating_add(minute.saturating_mul(60))
|
||||||
|
.saturating_add(second),
|
||||||
|
)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -276,6 +276,22 @@ impl OfficialLocalManifestAuditReport {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Existing local state for an official pull plan.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub struct OfficialLocalResourceState {
|
||||||
|
/// Number of expected URLs that already have download-manifest entries.
|
||||||
|
pub manifest_entry_count: usize,
|
||||||
|
/// Number of expected URL destinations that already exist on disk.
|
||||||
|
pub existing_file_count: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl OfficialLocalResourceState {
|
||||||
|
/// Returns true when this plan has any reusable or auditable local state.
|
||||||
|
pub fn has_any_resources(self) -> bool {
|
||||||
|
self.manifest_entry_count > 0 || self.existing_file_count > 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Executes an official pull plan with the system `curl` command.
|
/// Executes an official pull plan with the system `curl` command.
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct OfficialResourcePullService {
|
pub struct OfficialResourcePullService {
|
||||||
@@ -357,6 +373,9 @@ impl OfficialResourcePullService {
|
|||||||
let urls = plan.all_urls()?;
|
let urls = plan.all_urls()?;
|
||||||
let total = urls.len();
|
let total = urls.len();
|
||||||
let mut items = Vec::new();
|
let mut items = Vec::new();
|
||||||
|
let mut verified_hashes = Vec::new();
|
||||||
|
let mut verified_hash_urls = HashSet::<String>::new();
|
||||||
|
let mut processed_urls = HashSet::<String>::new();
|
||||||
for (offset, url) in urls.into_iter().enumerate() {
|
for (offset, url) in urls.into_iter().enumerate() {
|
||||||
if should_cancel() {
|
if should_cancel() {
|
||||||
return Err("official resource pull interrupted by shutdown request".to_string());
|
return Err("official resource pull interrupted by shutdown request".to_string());
|
||||||
@@ -406,6 +425,14 @@ impl OfficialResourcePullService {
|
|||||||
result.bytes,
|
result.bytes,
|
||||||
result.transferred_bytes,
|
result.transferred_bytes,
|
||||||
));
|
));
|
||||||
|
processed_urls.insert(url.clone());
|
||||||
|
self.verify_ready_official_hashes(
|
||||||
|
&official_hash_pairs,
|
||||||
|
&processed_urls,
|
||||||
|
&mut verified_hash_urls,
|
||||||
|
&mut verified_hashes,
|
||||||
|
&mut manifest,
|
||||||
|
)?;
|
||||||
items.push(OfficialResourcePullItem {
|
items.push(OfficialResourcePullItem {
|
||||||
url,
|
url,
|
||||||
destination,
|
destination,
|
||||||
@@ -418,8 +445,7 @@ impl OfficialResourcePullService {
|
|||||||
if should_cancel() {
|
if should_cancel() {
|
||||||
return Err("official resource pull interrupted by shutdown request".to_string());
|
return Err("official resource pull interrupted by shutdown request".to_string());
|
||||||
}
|
}
|
||||||
|
self.verify_all_official_hashes_are_complete(&official_hash_pairs, &verified_hash_urls)?;
|
||||||
let verified_hashes = self.verify_downloaded_official_hashes(&official_hash_pairs)?;
|
|
||||||
|
|
||||||
Ok(OfficialResourcePullReport {
|
Ok(OfficialResourcePullReport {
|
||||||
items,
|
items,
|
||||||
@@ -447,6 +473,34 @@ impl OfficialResourcePullService {
|
|||||||
Ok(OfficialLocalManifestAuditReport { items })
|
Ok(OfficialLocalManifestAuditReport { items })
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Returns whether the current output root already contains local state for
|
||||||
|
/// this plan. This is intentionally cheaper than a full audit and is used
|
||||||
|
/// to choose between audit-then-repair and first-time full pull flows.
|
||||||
|
pub fn local_resource_state(
|
||||||
|
&self,
|
||||||
|
plan: &OfficialResourcePullPlan,
|
||||||
|
) -> Result<OfficialLocalResourceState, String> {
|
||||||
|
let manifest = self.read_download_manifest()?;
|
||||||
|
let mut manifest_entry_count = 0;
|
||||||
|
let mut existing_file_count = 0;
|
||||||
|
|
||||||
|
for url in plan.all_urls()? {
|
||||||
|
if manifest.entries.contains_key(&url) {
|
||||||
|
manifest_entry_count += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
let destination = self.destination_for_url(&url)?;
|
||||||
|
if file_len_if_exists(&destination)?.is_some() {
|
||||||
|
existing_file_count += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(OfficialLocalResourceState {
|
||||||
|
manifest_entry_count,
|
||||||
|
existing_file_count,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/// Fetches an official URL into memory.
|
/// Fetches an official URL into memory.
|
||||||
///
|
///
|
||||||
/// This is intended for small discovery inputs such as `server-info`,
|
/// This is intended for small discovery inputs such as `server-info`,
|
||||||
@@ -604,37 +658,103 @@ impl OfficialResourcePullService {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn verify_downloaded_official_hashes(
|
fn verify_ready_official_hashes(
|
||||||
&self,
|
&self,
|
||||||
pairs: &[OfficialSeedHashPair],
|
pairs: &[OfficialSeedHashPair],
|
||||||
) -> Result<Vec<OfficialResourceHashVerification>, String> {
|
processed_urls: &HashSet<String>,
|
||||||
let mut verified = Vec::new();
|
verified_hash_urls: &mut HashSet<String>,
|
||||||
|
verified_hashes: &mut Vec<OfficialResourceHashVerification>,
|
||||||
|
manifest: &mut OfficialDownloadManifest,
|
||||||
|
) -> Result<(), String> {
|
||||||
for pair in pairs {
|
for pair in pairs {
|
||||||
let data_path = self.destination_for_url(&pair.data_url)?;
|
if verified_hash_urls.contains(&pair.hash_url) {
|
||||||
let hash_path = self.destination_for_url(&pair.hash_url)?;
|
continue;
|
||||||
let data = fs::read(&data_path).map_err(|error| {
|
}
|
||||||
format!(
|
|
||||||
"Failed to read official hash data file {}: {error}",
|
|
||||||
data_path.display()
|
|
||||||
)
|
|
||||||
})?;
|
|
||||||
let hash = fs::read(&hash_path).map_err(|error| {
|
|
||||||
format!(
|
|
||||||
"Failed to read official hash sidecar {}: {error}",
|
|
||||||
hash_path.display()
|
|
||||||
)
|
|
||||||
})?;
|
|
||||||
|
|
||||||
verified.push(verify_official_seed_catalog_hash(
|
if !processed_urls.contains(&pair.data_url) || !processed_urls.contains(&pair.hash_url)
|
||||||
&pair.data_url,
|
{
|
||||||
&data,
|
continue;
|
||||||
&pair.hash_url,
|
}
|
||||||
&hash,
|
|
||||||
)?);
|
if !self.hash_pair_files_exist(pair)? {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let verification = match self.verify_official_seed_hash_pair_from_disk(pair) {
|
||||||
|
Ok(verification) => verification,
|
||||||
|
Err(error) => {
|
||||||
|
if let Err(cleanup_error) =
|
||||||
|
self.invalidate_download_manifest_hash_pair(manifest, pair)
|
||||||
|
{
|
||||||
|
return Err(format!(
|
||||||
|
"{error}; additionally failed to invalidate local manifest entries for official hash pair: {cleanup_error}"
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
return Err(error);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
verified_hashes.push(verification);
|
||||||
|
verified_hash_urls.insert(pair.hash_url.clone());
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(verified)
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn hash_pair_files_exist(&self, pair: &OfficialSeedHashPair) -> Result<bool, String> {
|
||||||
|
let data_path = self.destination_for_url(&pair.data_url)?;
|
||||||
|
let hash_path = self.destination_for_url(&pair.hash_url)?;
|
||||||
|
Ok(file_len_if_exists(&data_path)?.is_some() && file_len_if_exists(&hash_path)?.is_some())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn verify_official_seed_hash_pair_from_disk(
|
||||||
|
&self,
|
||||||
|
pair: &OfficialSeedHashPair,
|
||||||
|
) -> Result<OfficialResourceHashVerification, String> {
|
||||||
|
let data_path = self.destination_for_url(&pair.data_url)?;
|
||||||
|
let hash_path = self.destination_for_url(&pair.hash_url)?;
|
||||||
|
let data = fs::read(&data_path).map_err(|error| {
|
||||||
|
format!(
|
||||||
|
"Failed to read official hash data file {}: {error}",
|
||||||
|
data_path.display()
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
let hash = fs::read(&hash_path).map_err(|error| {
|
||||||
|
format!(
|
||||||
|
"Failed to read official hash sidecar {}: {error}",
|
||||||
|
hash_path.display()
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
verify_official_seed_catalog_hash(&pair.data_url, &data, &pair.hash_url, &hash)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn invalidate_download_manifest_hash_pair(
|
||||||
|
&self,
|
||||||
|
manifest: &mut OfficialDownloadManifest,
|
||||||
|
pair: &OfficialSeedHashPair,
|
||||||
|
) -> Result<(), String> {
|
||||||
|
manifest.entries.remove(&pair.data_url);
|
||||||
|
manifest.entries.remove(&pair.hash_url);
|
||||||
|
self.write_download_manifest(manifest)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn verify_all_official_hashes_are_complete(
|
||||||
|
&self,
|
||||||
|
pairs: &[OfficialSeedHashPair],
|
||||||
|
verified_hash_urls: &HashSet<String>,
|
||||||
|
) -> Result<(), String> {
|
||||||
|
for pair in pairs {
|
||||||
|
if !verified_hash_urls.contains(&pair.hash_url) {
|
||||||
|
return Err(format!(
|
||||||
|
"official hash pair was not verified for {} using {}",
|
||||||
|
pair.data_url, pair.hash_url
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn read_download_manifest(&self) -> Result<OfficialDownloadManifest, String> {
|
fn read_download_manifest(&self) -> Result<OfficialDownloadManifest, String> {
|
||||||
@@ -1607,6 +1727,43 @@ fi
|
|||||||
.all(|item| item.status == OfficialResourcePullStatus::Downloaded));
|
.all(|item| item.status == OfficialResourcePullStatus::Downloaded));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn local_resource_state_reports_manifest_entries_and_existing_files() {
|
||||||
|
let out_dir = TempDir::new().unwrap();
|
||||||
|
let bin_dir = TempDir::new().unwrap();
|
||||||
|
let curl_path = bin_dir.path().join("curl");
|
||||||
|
write_fake_curl(&curl_path);
|
||||||
|
|
||||||
|
let service = OfficialResourcePullService::with_curl_command(out_dir.path(), &curl_path);
|
||||||
|
let plan = build_official_pull_plan(discovery_plan(), inventory());
|
||||||
|
let urls = plan.all_urls().unwrap();
|
||||||
|
|
||||||
|
let empty_state = service.local_resource_state(&plan).unwrap();
|
||||||
|
assert_eq!(empty_state.manifest_entry_count, 0);
|
||||||
|
assert_eq!(empty_state.existing_file_count, 0);
|
||||||
|
assert!(!empty_state.has_any_resources());
|
||||||
|
|
||||||
|
let first_destination = service.destination_for_url(&urls[0]).unwrap();
|
||||||
|
fs::create_dir_all(first_destination.parent().unwrap()).unwrap();
|
||||||
|
fs::write(&first_destination, b"local-only").unwrap();
|
||||||
|
|
||||||
|
let file_only_state = service.local_resource_state(&plan).unwrap();
|
||||||
|
assert_eq!(file_only_state.manifest_entry_count, 0);
|
||||||
|
assert_eq!(file_only_state.existing_file_count, 1);
|
||||||
|
assert!(file_only_state.has_any_resources());
|
||||||
|
|
||||||
|
let mut manifest = OfficialDownloadManifest::default();
|
||||||
|
service
|
||||||
|
.record_download_manifest_entry(&mut manifest, &urls[0], &first_destination)
|
||||||
|
.unwrap();
|
||||||
|
service.write_download_manifest(&manifest).unwrap();
|
||||||
|
|
||||||
|
let manifest_state = service.local_resource_state(&plan).unwrap();
|
||||||
|
assert_eq!(manifest_state.manifest_entry_count, 1);
|
||||||
|
assert_eq!(manifest_state.existing_file_count, 1);
|
||||||
|
assert!(manifest_state.has_any_resources());
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn redownloads_existing_file_when_manifest_hash_mismatches() {
|
fn redownloads_existing_file_when_manifest_hash_mismatches() {
|
||||||
let out_dir = TempDir::new().unwrap();
|
let out_dir = TempDir::new().unwrap();
|
||||||
@@ -1677,6 +1834,17 @@ fi
|
|||||||
|
|
||||||
let error = service.pull(&plan).unwrap_err();
|
let error = service.pull(&plan).unwrap_err();
|
||||||
assert!(error.contains("official hash mismatch"));
|
assert!(error.contains("official hash mismatch"));
|
||||||
|
|
||||||
|
let manifest = service.read_download_manifest().unwrap();
|
||||||
|
assert!(manifest.entries.is_empty());
|
||||||
|
|
||||||
|
let audit = service.audit_local_manifest(&plan).unwrap();
|
||||||
|
assert!(!audit.is_clean());
|
||||||
|
assert_eq!(audit.repair_needed_count(), 2);
|
||||||
|
assert!(audit
|
||||||
|
.items
|
||||||
|
.iter()
|
||||||
|
.all(|item| { item.status == OfficialLocalManifestAuditStatus::MissingManifestEntry }));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
@@ -627,35 +627,39 @@ impl OfficialUpdateService {
|
|||||||
sync_plan.decision, config.force, remote_should_download
|
sync_plan.decision, config.force, remote_should_download
|
||||||
),
|
),
|
||||||
));
|
));
|
||||||
let pull_plan = if config.audit_local || remote_should_download || config.plan {
|
progress(OfficialUpdateProgress::new(
|
||||||
progress(OfficialUpdateProgress::new(
|
"plan",
|
||||||
"plan",
|
"building official pull plan from latest seed catalogs",
|
||||||
"building official pull plan from seed catalogs",
|
));
|
||||||
));
|
let pull_plan = build_pull_plan(
|
||||||
Some(build_pull_plan(
|
&server_info,
|
||||||
&server_info,
|
&connection_group,
|
||||||
&connection_group,
|
&app_version,
|
||||||
&app_version,
|
platforms,
|
||||||
platforms,
|
&fetcher,
|
||||||
&fetcher,
|
&mut progress,
|
||||||
&mut progress,
|
&mut should_cancel,
|
||||||
&mut should_cancel,
|
)?;
|
||||||
)?)
|
let url_count = pull_plan.all_urls().map_err(anyhow::Error::msg)?.len();
|
||||||
} else {
|
progress(OfficialUpdateProgress::new(
|
||||||
progress(OfficialUpdateProgress::new(
|
"plan",
|
||||||
"plan",
|
format!("pull plan contains {url_count} official URLs"),
|
||||||
"pull plan not needed for clean remote state and disabled plan output",
|
));
|
||||||
));
|
let local_state = fetcher
|
||||||
None
|
.local_resource_state(&pull_plan)
|
||||||
};
|
.map_err(anyhow::Error::msg)?;
|
||||||
if let Some(pull_plan) = pull_plan.as_ref() {
|
let has_local_resources = local_state.has_any_resources();
|
||||||
let url_count = pull_plan.all_urls().map_err(anyhow::Error::msg)?.len();
|
progress(OfficialUpdateProgress::new(
|
||||||
progress(OfficialUpdateProgress::new(
|
"local-state",
|
||||||
"plan",
|
format!(
|
||||||
format!("pull plan contains {url_count} official URLs"),
|
"manifest_entries={} existing_files={} has_local_resources={}",
|
||||||
));
|
local_state.manifest_entry_count,
|
||||||
}
|
local_state.existing_file_count,
|
||||||
let local_audit = if config.audit_local {
|
has_local_resources
|
||||||
|
),
|
||||||
|
));
|
||||||
|
let should_audit_local = config.audit_local && has_local_resources;
|
||||||
|
let local_audit = if should_audit_local {
|
||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
"audit",
|
"audit",
|
||||||
format!(
|
format!(
|
||||||
@@ -665,13 +669,15 @@ impl OfficialUpdateService {
|
|||||||
));
|
));
|
||||||
Some(
|
Some(
|
||||||
fetcher
|
fetcher
|
||||||
.audit_local_manifest(
|
.audit_local_manifest(&pull_plan)
|
||||||
pull_plan
|
|
||||||
.as_ref()
|
|
||||||
.ok_or_else(|| anyhow::anyhow!("local audit requires a pull plan"))?,
|
|
||||||
)
|
|
||||||
.map_err(anyhow::Error::msg)?,
|
.map_err(anyhow::Error::msg)?,
|
||||||
)
|
)
|
||||||
|
} else if config.audit_local {
|
||||||
|
progress(OfficialUpdateProgress::new(
|
||||||
|
"audit",
|
||||||
|
"no local resources found; skipping local audit before first full pull",
|
||||||
|
));
|
||||||
|
None
|
||||||
} else {
|
} else {
|
||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
"audit",
|
"audit",
|
||||||
@@ -692,7 +698,8 @@ impl OfficialUpdateService {
|
|||||||
.as_ref()
|
.as_ref()
|
||||||
.map(|audit| !audit.is_clean())
|
.map(|audit| !audit.is_clean())
|
||||||
.unwrap_or(false);
|
.unwrap_or(false);
|
||||||
let should_download = remote_should_download || repair_needed;
|
let initial_pull_needed = !has_local_resources;
|
||||||
|
let should_download = remote_should_download || repair_needed || initial_pull_needed;
|
||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
"audit",
|
"audit",
|
||||||
format!(
|
format!(
|
||||||
@@ -703,8 +710,8 @@ impl OfficialUpdateService {
|
|||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
"decision",
|
"decision",
|
||||||
format!(
|
format!(
|
||||||
"final should_download={} repair_needed={}",
|
"final should_download={} repair_needed={} initial_pull_needed={}",
|
||||||
should_download, repair_needed
|
should_download, repair_needed, initial_pull_needed
|
||||||
),
|
),
|
||||||
));
|
));
|
||||||
check_shutdown_requested(&mut should_cancel)?;
|
check_shutdown_requested(&mut should_cancel)?;
|
||||||
@@ -760,13 +767,7 @@ impl OfficialUpdateService {
|
|||||||
|
|
||||||
if config.dry_run {
|
if config.dry_run {
|
||||||
if config.plan {
|
if config.plan {
|
||||||
let urls = pull_plan
|
let urls = pull_plan.all_urls().map_err(anyhow::Error::msg)?;
|
||||||
.as_ref()
|
|
||||||
.ok_or_else(|| {
|
|
||||||
anyhow::anyhow!("dry-run plan requested but no pull plan exists")
|
|
||||||
})?
|
|
||||||
.all_urls()
|
|
||||||
.map_err(anyhow::Error::msg)?;
|
|
||||||
report.download_url_count = Some(urls.len());
|
report.download_url_count = Some(urls.len());
|
||||||
report.download_urls = urls;
|
report.download_urls = urls;
|
||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
@@ -791,9 +792,6 @@ impl OfficialUpdateService {
|
|||||||
}
|
}
|
||||||
check_shutdown_requested(&mut should_cancel)?;
|
check_shutdown_requested(&mut should_cancel)?;
|
||||||
|
|
||||||
let pull_plan = pull_plan
|
|
||||||
.as_ref()
|
|
||||||
.ok_or_else(|| anyhow::anyhow!("download requested but no pull plan exists"))?;
|
|
||||||
let download_url_count = pull_plan.all_urls().map_err(anyhow::Error::msg)?.len();
|
let download_url_count = pull_plan.all_urls().map_err(anyhow::Error::msg)?.len();
|
||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
"download",
|
"download",
|
||||||
@@ -801,7 +799,7 @@ impl OfficialUpdateService {
|
|||||||
));
|
));
|
||||||
let pull_report = fetcher
|
let pull_report = fetcher
|
||||||
.pull_with_progress_and_cancellation(
|
.pull_with_progress_and_cancellation(
|
||||||
pull_plan,
|
&pull_plan,
|
||||||
|event| {
|
|event| {
|
||||||
progress(progress_from_pull_event(event));
|
progress(progress_from_pull_event(event));
|
||||||
},
|
},
|
||||||
@@ -824,7 +822,7 @@ impl OfficialUpdateService {
|
|||||||
));
|
));
|
||||||
check_shutdown_requested(&mut should_cancel)?;
|
check_shutdown_requested(&mut should_cancel)?;
|
||||||
let final_audit = fetcher
|
let final_audit = fetcher
|
||||||
.audit_local_manifest(pull_plan)
|
.audit_local_manifest(&pull_plan)
|
||||||
.map_err(anyhow::Error::msg)?;
|
.map_err(anyhow::Error::msg)?;
|
||||||
progress(OfficialUpdateProgress::new(
|
progress(OfficialUpdateProgress::new(
|
||||||
"snapshot",
|
"snapshot",
|
||||||
|
|||||||
@@ -185,6 +185,58 @@ fn official_update_service_emits_human_progress_events() {
|
|||||||
.contains("pull plan contains 19 official URLs")));
|
.contains("pull plan contains 19 official URLs")));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn official_update_first_run_pulls_without_pre_audit() {
|
||||||
|
let harness = TestHarness::new();
|
||||||
|
let config = harness.sync_config("first-run-output");
|
||||||
|
let mut events = Vec::<OfficialUpdateProgress>::new();
|
||||||
|
|
||||||
|
let report = OfficialUpdateService::new()
|
||||||
|
.run_with_progress(&config, |event| events.push(event))
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
assert_eq!(report.update_status, OfficialUpdateStatus::Downloaded);
|
||||||
|
assert_eq!(report.resource_count, Some(19));
|
||||||
|
assert_eq!(report.downloaded_count, 19);
|
||||||
|
assert_eq!(report.skipped_count, 0);
|
||||||
|
assert_eq!(report.local_manifest_repair_needed_count, 0);
|
||||||
|
assert!(events.iter().any(|event| {
|
||||||
|
event.stage == "local-state" && event.message.contains("has_local_resources=false")
|
||||||
|
}));
|
||||||
|
assert!(events
|
||||||
|
.iter()
|
||||||
|
.any(|event| { event.stage == "audit" && event.message.contains("skipping local audit") }));
|
||||||
|
assert!(events.iter().any(|event| {
|
||||||
|
event.stage == "decision" && event.message.contains("initial_pull_needed=true")
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn official_update_second_run_audits_existing_resources_before_reuse() {
|
||||||
|
let harness = TestHarness::new();
|
||||||
|
let config = harness.sync_config("second-run-output");
|
||||||
|
OfficialUpdateService::new().run(&config).unwrap();
|
||||||
|
|
||||||
|
let mut events = Vec::<OfficialUpdateProgress>::new();
|
||||||
|
let report = OfficialUpdateService::new()
|
||||||
|
.run_with_progress(&config, |event| events.push(event))
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
assert_eq!(report.update_status, OfficialUpdateStatus::UpToDate);
|
||||||
|
assert_eq!(report.downloaded_count, 0);
|
||||||
|
assert_eq!(report.local_manifest_verified_count, 19);
|
||||||
|
assert_eq!(report.local_manifest_repair_needed_count, 0);
|
||||||
|
assert!(events.iter().any(|event| {
|
||||||
|
event.stage == "local-state" && event.message.contains("has_local_resources=true")
|
||||||
|
}));
|
||||||
|
assert!(events.iter().any(|event| {
|
||||||
|
event.stage == "audit" && event.message.contains("auditing local download manifest")
|
||||||
|
}));
|
||||||
|
assert!(events.iter().any(|event| {
|
||||||
|
event.stage == "decision" && event.message.contains("initial_pull_needed=false")
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
struct TestHarness {
|
struct TestHarness {
|
||||||
temp: TempDir,
|
temp: TempDir,
|
||||||
curl_script: std::path::PathBuf,
|
curl_script: std::path::PathBuf,
|
||||||
@@ -259,6 +311,17 @@ impl TestHarness {
|
|||||||
&self.curl_script,
|
&self.curl_script,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn sync_config(&self, output_name: &str) -> OfficialUpdateConfig {
|
||||||
|
OfficialUpdateConfig {
|
||||||
|
auto_discover: true,
|
||||||
|
launcher_version: TEST_LAUNCHER_VERSION.to_string(),
|
||||||
|
output_root: self.temp.path().join(output_name),
|
||||||
|
curl_command: self.curl_script.clone(),
|
||||||
|
unzip_command: self.unzip_script.clone(),
|
||||||
|
..OfficialUpdateConfig::default()
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn write_executable(path: &Path, content: &str) {
|
fn write_executable(path: &Path, content: &str) {
|
||||||
|
|||||||
Reference in New Issue
Block a user