diff --git a/CURRENT_STATUS.md b/CURRENT_STATUS.md index 0f191bb..da1e62b 100644 --- a/CURRENT_STATUS.md +++ b/CURRENT_STATUS.md @@ -113,7 +113,7 @@ TextUnit scope、source history 和 approved review。worker、TM 复用、人 待完成: -- `crates/bat-assetbundle` 已具备 UnityFS 容器、对象表、TypeTree 元数据、基础字段读取、TextAsset 和 TextUnit 提取;UnityFS 容器已补充总大小、计数、路径、重复 directory、LZMA 和边界校验,并通过 UnityPy 真实 bundle 隔离回归;已有 TextAsset、TypeTree string field 和 managed-reference string field 的 localized patch 发布闭环,真实复杂版本差异、整体 AssetBundle 重打包和通用 Patch 仍未实现。 +- `crates/bat-assetbundle` 已具备 UnityFS 容器、对象表、TypeTree 元数据、基础字段读取、TextAsset 和 TextUnit 提取;UnityFS 容器已补充总大小、计数、路径、重复 directory、LZMA 和边界校验,并通过 UnityPy 真实 bundle 隔离回归。对当前真实/合成回归覆盖的结构,TextAsset、TypeTree string field、managed-reference string field 和语义字段已形成 parse→modify→rebuild→reparse 闭环,重建保留已识别压缩/对齐/目录形态并校验未修改对象/字段;整体任意 AssetBundle、未知结构和全部真实版本差异仍未实现。 - Addressables parser 已覆盖当前真实形态 fixture/golden 与 `m_Crc`,但仍需继续覆盖二进制/压缩字段组合和更细失败诊断。 - 客户端发现、备份、应用补丁流程尚未连接真实实现。 @@ -162,7 +162,7 @@ TextUnit scope、source history 和 approved review。worker、TM 复用、人 ### `bat-assetbundle` -状态:**UnityFS 解包、TypeTree 字段读取、TextUnit 提取和受支持 localized patch 发布已可用;复杂结构覆盖与整体 AssetBundle 重打包仍待继续补齐** +状态:**已验证 UnityFS 结构的解析、变长修改、重建、重解析和受支持 localized 发布可用;任意复杂结构兼容仍待继续补齐** 解析扩展当前按路线图和真实 fixture 验收推进。 @@ -181,7 +181,7 @@ TextUnit scope、source history 和 approved review。worker、TM 复用、人 待完成: - 真实 MonoBehaviour、ScriptableObject 版本差异、复杂容器结构调整、unknown 字段结构语义和未见样本驱动的完整 managed reference registry / map entry 变体覆盖;TypeTree-covered managed reference 字段与 registry 记录已可结构化解码,常见 full typename 可拆解为 assembly/namespace/class,不做低保真猜测。 -- 复杂对象整体结构修改后的发布级 AssetBundle 重打包;UnityFS TextAsset、TypeTree string 字段、managed-reference registry payload 字符串、基础语义字段、enum、bit_field、object 字段组合和 TypeTree schema 支撑的 array/vector/map 整体替换的文件级链路已具备重建后校验,受支持 localized patch 已有独立 staging、manifest、current、状态校验和显式 rollback;整体 AssetBundle 发布仍未完成。 +- 任意复杂对象整体结构和所有真实版本差异的发布级 AssetBundle 重打包;当前已验证的 UnityFS TextAsset、TypeTree string 字段、managed-reference registry payload 字符串、基础语义字段、enum、bit_field、object 字段组合和 TypeTree schema 支撑的 array/vector/map 整体替换已具备变长重建、压缩/对齐保留、未修改对象/字段校验、受支持 localized staging/manifest/current/rollback;带 `archive_entry` 的可验证 ZIP 内 bundle 也会重写外层 ZIP。 - 真实资源 fixture 覆盖对象级解析和文本提取。 - 详细补全顺序见 `docs/architecture/assetbundle.md`。 @@ -194,7 +194,7 @@ TextUnit scope、source history 和 approved review。worker、TM 复用、人 待完成: - 未见样本驱动的 map entry schema 变化、unknown 字段结构语义、完整 managed reference registry 变体驱动字段修改后的语义重打包。 -- 通用 manifest 驱动的跨类型 patch build/apply/diff 发布;当前 localized 发布仅接受已验证 TextUnit 对应的受支持 UnityFS 文本字段,并不等价于整体 AssetBundle 重打包。 +- 通用 manifest 驱动的跨类型 patch build/apply/diff 发布;当前 localized 发布仍仅接受已验证 TextUnit 对应的受支持 UnityFS 文本字段,不等价于任意整体 AssetBundle 重打包。 - `unityfs.inspect`、复杂 UnityFS 语义编辑和写入型发布工作流仍未开放。 ### `bat-ffi` diff --git a/PROJECT_PLAN.md b/PROJECT_PLAN.md index 2da7b09..5ab6146 100644 --- a/PROJECT_PLAN.md +++ b/PROJECT_PLAN.md @@ -41,12 +41,12 @@ BlueArchiveToolkit 不是一次性脚本,也不是演示项目。最终交付 ### 仍是骨架或占位 -1. `bat-assetbundle` 已具备 UnityFS 解包和 TextAsset 提取基础能力(header/block info/directory、LZ4/LZMA block info 与数据 block、directory 文件提取、serialized file object table、TypeTree node 元数据、TextAsset bytes、TypeTree-covered managed reference payload TextUnit 上下文),并已有受支持 localized patch 发布能力;MonoBehaviour/ScriptableObject 复杂字段级解析、整体重打包和通用 Patch 仍未完成。 -2. `bat-patch` 已具备确定性 Binary hunk diff/apply、RFC 6902 JSON Patch apply、UTF-8 Text Patch、通用 Patch manifest、BLAKE3/size 完整性校验和 rollback 元数据;文件级 `patch.apply` RPC / `patch-apply` CLI 与 UnityFS TextAsset / TypeTree string / TypeTree 语义字段写入入口已开放,`bat-assetbundle` + `LocalizedPatchService` 已完成受支持 TextUnit 到 localized patch manifest、独立 staging、发布和 rollback 闭环,通用 manifest 发布与整体 AssetBundle 重打包仍后置。 +1. `bat-assetbundle` 已具备 UnityFS 解包、TextAsset/TypeTree 字段读取和 TextUnit 提取;对当前真实/合成回归覆盖的结构,UnityFS TextAsset、TypeTree string、managed-reference string 和语义字段已形成 parse→modify→rebuild→reparse 闭环,保留已识别压缩/对齐/目录形态并校验未修改对象/字段;所有真实版本差异、未知字段语义和任意复杂 AssetBundle 兼容仍未完成。 +2. `bat-patch` 已具备确定性 Binary hunk diff/apply、RFC 6902 JSON Patch apply、UTF-8 Text Patch、通用 Patch manifest、BLAKE3/size 完整性校验和 rollback 元数据;文件级 `patch.apply` RPC / `patch-apply` CLI 与 UnityFS TextAsset / TypeTree string / TypeTree 语义字段写入入口已开放,`bat-assetbundle` + `LocalizedPatchService` 已完成受支持 TextUnit 到 localized patch manifest、独立 staging、发布和 rollback 闭环;ZIP 内 bundle 在 `archive_entry` 可验证时会重写外层 ZIP,通用 manifest 发布与任意 AssetBundle 重打包仍后置。 3. Go 侧边界已确定(见 `docs/reports/GO_STATUS.md`):同步/运维命令行 = Rust `bat`;资源分发和内嵌 dashboard = `cmd/bat-api` MVP;`internal/backendrpc` 完成;`cmd/bat` 仅为试验(`bin/bat-go`)。完整游戏业务 API / 完整 Web 协作后台 / SDK 仍未完成。 4. Addressables parser 已覆盖当前真实形态 fixture/golden,但还不是完整 Unity Addressables/SBP catalog 兼容层。 5. 官方同步结果可配置为发布后自动导入 CAS + ResourceRepository,并通过 `resource.index` RPC/CLI 查询;Resource metadata 已保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 摘要,资源级查询已覆盖 release、平台、destination、archive entry、parse status 和 TextUnit format;单条 TextUnit 明细和解析错误已持久化到 `official-textunit-index.json`,可通过 `parse.text_units` / `parse.errors` 查询;离线 TextUnit 翻译任务状态和跳过/失败原因可通过 `translation.tasks` 查询,`translation.task.update` 已提供 worker 状态回写 contract,`translation.worker.run` 已提供真实 provider worker 触发、lease/retry 和结果落库 contract,`translation.proofread` 已提供汉化 workflow 人工校对标记 contract,`translation.memory.*` 已提供 Rust-owned TM 摘要、raw source/context 查询、provenance 和显式 confirm contract,Go 侧仅代理。 -6. 受支持汉化 Patch 发布已具备 UnityFS TextAsset、TypeTree string field 和 managed-reference string field 的 manifest/apply/rollback/完整性校验和 `localized.status` 严格校验;真实 provider worker 与项目级 Translation Memory V1 已接入,翻译记忆到完整汉化文件集合的构建仍未完成。 +6. 受支持汉化 Patch 发布已具备 UnityFS TextAsset、TypeTree string field 和 managed-reference string field 的 manifest/apply/rollback/完整性校验和 `localized.status` 严格校验;ZIP 内 bundle 在 `archive_entry` 可验证时会重写外层 ZIP。真实 provider worker 与项目级 Translation Memory V1 已接入,翻译记忆到完整汉化文件集合的构建仍未完成。 7. 真实官方网络全量下载 smoke 已固化为可重复脚本和 runbook;真实运行记录处于长期运行测试阶段,报告待后续提供。 8. 内嵌 dashboard MVP 已实现;完整 Web 协作后台、数据库迁移、插件加载机制尚未实现;`bat-api` 资源 bootstrap/分发/OpenAPI/管理控制面已通过 `/openapi.yaml` 提供,完整游戏业务 API 的 OpenAPI 仍未完成。 9. 原 Git 历史未恢复;当前仓库以新初始化基线为准。 @@ -213,7 +213,7 @@ BlueArchiveToolkit 不是一次性脚本,也不是演示项目。最终交付 1. **解析缓存闭环**:官方同步发布后生成 `official-parse-cache.json`,覆盖 manifest 全部条目、直接 bundle、zip 内 bundle、非候选资源和解析失败诊断;未变化文件按 URL、相对路径、size 和 BLAKE3 复用解析结果。 2. **Addressables 完整化**:覆盖 Windows/Android JSON、compact JSON 和后续二进制 catalog 入口,解析 provider、internal id、primary key、dependency、bundle name、hash、size、CRC 和资源类型。 -3. **UnityFS 容器层**:基础目标已完成 header、block info、directory、data block、LZ4/LZMA、alignment、总大小/计数/路径/边界错误、directory 文件提取和 UnityPy 真实样本回归;复杂版本差异和发布级重打包另行推进。 +3. **UnityFS 容器层**:基础目标已完成 header、block info、directory、data block、LZ4/LZMA、alignment、总大小/计数/路径/边界错误、directory 文件提取和 UnityPy 真实样本回归;当前已验证结构另有压缩/对齐保留的变长发布级重建闭环,复杂版本差异和任意结构重打包另行推进。 4. **Serialized file 层**:稳定 Unity serialized file header、type table、TypeTree node、object table、path id、class id 和 raw object bytes 表示。 5. **字段级解析层**:实现 TypeTree 字段 reader,支持 bool、integer、float、string、bytes、array、vector/staticvector 嵌套 `Array`、`List` / `HashSet` 集合 alias、map、PPtr、enum `value__` backing field、`LayerMask` / `BitField` 的 `m_Bits` backing field、常见固定 Unity float/int/hash 值类型的 leaf/direct-child 形态、unknown fixed-size raw bytes 保留和同长度替换、TypeTree-covered managed reference / `SerializedReference` alias 和 TypeTree-covered managed reference registry 记录;managed-reference full typename 可拆为 assembly/namespace/class,常见 `m_ManagedReferences` / `RefIds` / verbose type 字段命名、`managedReference*` / `serializedReference*` metadata 和 `data` / `value` / `payload` / `object` / `managedReferencePayload` / `referencePayload` / `serializedReferencePayload` / `managedReferenceValue` / `referenceValue` / `serializedReferenceValue` / `managedReferenceObject` / `referenceObject` / `serializedReferenceObject` / `managedReferenceData` / `referenceData` / `serializedData` / `serializedReferenceData` payload 命名已有回归覆盖,TextUnit 只提取 payload 字符串并按结构化 record、`RefIds[n]` 等记录前缀或子字段保留类型上下文;array/vector/List/HashSet/map 元素与 registry payload 字段保留独立 field path、offset 和 byte size,可支撑字符串元素、managed-reference registry payload 字段、基础语义字段 patch、enum/bit_field 语义 patch、固定值类型 patch、unknown fixed-size bytes patch、object 字段组合和 TypeTree schema 支撑的 array/vector/List/HashSet/map 整体变长替换,`first/second` 与 `key/value` map entry schema 已有回归覆盖;解析模块当前仍有真实版本差异、未见样本驱动的完整 managed reference registry / map entry 变体和 unknown 字段结构语义需要继续推进。 6. **文本对象入口**:实现 TextAsset、MonoBehaviour、ScriptableObject 的可扩展提取入口,输出可追溯到 bundle、serialized file、path id 和 field path 的文本定位。 @@ -461,7 +461,7 @@ BlueArchiveToolkit 不是一次性脚本,也不是演示项目。最终交付 按最终目标计算,当前总体完成度不固定写单一百分比,以模块状态、源码、测试和契约为准。 -已完成的是稳定基线、架构骨架、部分接口、CAS V1、Rust 官方资源同步闭环、可配置 CAS/ResourceRepository 导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、provider worker、Translation Memory V1、Glossary V1、通用 Binary/JSON/Text Patch 基础、受支持 localized patch 发布/rollback,以及 Go `bat-api` 资源分发、内嵌 dashboard 和同机 live 联调。下一阶段的关键是 TM/Glossary 扩展、通用 manifest 发布、复杂 AssetBundle 解析/重打包和官方资源长期运行报告。 +已完成的是稳定基线、架构骨架、部分接口、CAS V1、Rust 官方资源同步闭环、可配置 CAS/ResourceRepository 导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、provider worker、Translation Memory V1、Glossary V1、通用 Binary/JSON/Text Patch 基础、已验证结构的 AssetBundle 变长重建、受支持 localized patch 发布/rollback,以及 Go `bat-api` 资源分发、内嵌 dashboard 和同机 live 联调。下一阶段的关键是 TM/Glossary 扩展、通用 manifest 发布、真实版本与复杂 AssetBundle 兼容和官方资源长期运行报告。 --- diff --git a/USERGUIDE.md b/USERGUIDE.md index 2bd6621..f803de5 100644 --- a/USERGUIDE.md +++ b/USERGUIDE.md @@ -248,6 +248,7 @@ curl -i -H 'Range: bytes=0-1023' \ | `--proxy ` | curl 代理覆盖(默认 `auto`,从环境变量检测)。scheme 支持 http/https/socks4/socks4a/socks5/socks5h | | `--no-proxy` | 强制直连 | | `--unzip ` | unzip 可执行文件(默认 `unzip`) | +| `--zip ` | zip 可执行文件(默认 `zip`) | | `--dry-run` | 不写同步状态 | | `--plan` | dry-run 时输出计划中的 URL | | `--force` | 强制下载/刷新 | @@ -294,7 +295,7 @@ curl -i -H 'Range: bytes=0-1023' \ - `config.toml` 的字段按职责分组:`[runtime]`、`[resource]`、`[localized]`、`[repository]`、`[network]`、`[translation.worker]`。 - 现有 `BAT_*` 环境变量仍然有效,可继续覆盖 `config.toml` 中的同名配置。 - `BAT_SKIP_ENV_FILE` 已废弃且不再影响启动。 -- 支持的环境变量:`BAT_OUTPUT`、`BAT_LOCALIZED_OUTPUT`、`BAT_STATE_DIR`、`BAT_AUTO_DISCOVER`、`BAT_WATCH`、`BAT_DAEMON`、`BAT_IMPORT_REPOSITORY`、`BAT_IMPORT_CAS_ROOT`、`BAT_IMPORT_RESOURCE_DB`、`BAT_PROXY`、`BAT_NO_PROXY`、`BAT_INTERVAL_SECONDS`、`BAT_ERROR_RETRY_SECONDS`、`BAT_APP_VERSION`、`BAT_CONNECTION_GROUP`、`BAT_LAUNCHER_VERSION`、`BAT_PLATFORMS`、`BAT_CURL`、`BAT_DOWNLOAD_CONCURRENCY`、`BAT_UNZIP`、`BAT_JSON`、`BAT_QUIET_UP_TO_DATE`、`BAT_TRANSLATION_PROVIDER`、`BAT_TRANSLATION_FIXTURE`、`BAT_TRANSLATION_MEMORY_PATH`、`BAT_GLOSSARY_PATH`、`BAT_TRANSLATION_CONCURRENCY`、`BAT_TRANSLATION_MAX_ATTEMPTS`、`BAT_TRANSLATION_LEASE_SECONDS`、`BAT_TRANSLATION_RETRY_BACKOFF_SECONDS`、`BAT_TRANSLATION_MAX_TASKS`、`BAT_TRANSLATION_WORKER_ID`;也可以直接写 `HTTPS_PROXY` 等通用环境变量(走现有代理自动检测)。布尔值支持 `1/0/true/false/yes/no/on/off`。 +- 支持的环境变量:`BAT_OUTPUT`、`BAT_LOCALIZED_OUTPUT`、`BAT_STATE_DIR`、`BAT_AUTO_DISCOVER`、`BAT_WATCH`、`BAT_DAEMON`、`BAT_IMPORT_REPOSITORY`、`BAT_IMPORT_CAS_ROOT`、`BAT_IMPORT_RESOURCE_DB`、`BAT_PROXY`、`BAT_NO_PROXY`、`BAT_INTERVAL_SECONDS`、`BAT_ERROR_RETRY_SECONDS`、`BAT_APP_VERSION`、`BAT_CONNECTION_GROUP`、`BAT_LAUNCHER_VERSION`、`BAT_PLATFORMS`、`BAT_CURL`、`BAT_DOWNLOAD_CONCURRENCY`、`BAT_UNZIP`、`BAT_ZIP`、`BAT_JSON`、`BAT_QUIET_UP_TO_DATE`、`BAT_TRANSLATION_PROVIDER`、`BAT_TRANSLATION_FIXTURE`、`BAT_TRANSLATION_MEMORY_PATH`、`BAT_GLOSSARY_PATH`、`BAT_TRANSLATION_CONCURRENCY`、`BAT_TRANSLATION_MAX_ATTEMPTS`、`BAT_TRANSLATION_LEASE_SECONDS`、`BAT_TRANSLATION_RETRY_BACKOFF_SECONDS`、`BAT_TRANSLATION_MAX_TASKS`、`BAT_TRANSLATION_WORKER_ID`;也可以直接写 `HTTPS_PROXY` 等通用环境变量(走现有代理自动检测)。布尔值支持 `1/0/true/false/yes/no/on/off`。 - `BAT_WATCH` / `BAT_DAEMON` 只对无子命令的 `bat` 生效(两者同时为 `1` 时 daemon 优先);命令行显式传入 `--watch` / `--daemon` / `--dry-run` 时运行模式设置让位。`status` / `verify` 等子命令不受它们影响。 - 已运行的 daemon 不会热读 `config.toml`;默认 `reload` 只唤醒后台重新发现和刷新。需要应用配置文件变更时,使用带显式启动参数的 `restart`/`reload`,或先 `stop` 再重新启动 daemon。 - `BAT_REDIS_URL` / `BAT_REDIS_PASSWORD` 为**预留键**:Redis 任务后端尚未接入,当前任务历史持久化在 `/bat-tasks.json`。 diff --git a/crates/bat-assetbundle/src/lib.rs b/crates/bat-assetbundle/src/lib.rs index a69e28a..5c1e3f8 100644 --- a/crates/bat-assetbundle/src/lib.rs +++ b/crates/bat-assetbundle/src/lib.rs @@ -17,8 +17,8 @@ pub mod types; pub use error::{AssetBundleError, Result}; pub use parser::{compression_from_flags, Parser, UnityFsParser}; pub use patch::{ - patch_unityfs_field, patch_unityfs_string_field, patch_unityfs_text_asset, FieldPatch, - StringFieldPatch, TextAssetPatch, + patch_unityfs_field, patch_unityfs_string_field, patch_unityfs_text_asset, + rebuild_unityfs_bundle, FieldPatch, StringFieldPatch, TextAssetPatch, }; pub use serialized::{ UnityManagedReferenceMetadata, UnityManagedReferenceRecord, UnitySerializedField, diff --git a/crates/bat-assetbundle/src/parser.rs b/crates/bat-assetbundle/src/parser.rs index 2c27588..0978fdf 100644 --- a/crates/bat-assetbundle/src/parser.rs +++ b/crates/bat-assetbundle/src/parser.rs @@ -174,6 +174,7 @@ fn parse_unityfs(data: &[u8]) -> Result { compressed_data_size, uncompressed_data_size, raw_data: data.to_vec(), + uncompressed_data, files, serialized_files, text_assets, diff --git a/crates/bat-assetbundle/src/patch.rs b/crates/bat-assetbundle/src/patch.rs index 5bd6ea4..9c5148c 100644 --- a/crates/bat-assetbundle/src/patch.rs +++ b/crates/bat-assetbundle/src/patch.rs @@ -5,7 +5,7 @@ use crate::parser::UnityFsParser; use crate::serialized::{ UnitySerializedField, UnitySerializedReplacementValue, UnitySerializedValue, }; -use crate::types::UnityFsBundle; +use crate::types::{UnityFsBundle, UnityFsCompression}; use md5::{Digest, Md5}; /// One TextAsset replacement inside a serialized UnityFS directory file. @@ -105,9 +105,9 @@ impl FieldPatch { /// Patches one TextAsset and rebuilds the UnityFS container. /// -/// The rebuilt bundle uses a single uncompressed data block. This keeps the -/// patch path deterministic and avoids relying on a compressor-specific -/// implementation while preserving all directory file paths and metadata. +/// The rebuilt bundle retains the parsed block count, compression modes, +/// alignment flags and directory metadata while recalculating all variable +/// offsets and sizes. pub fn patch_unityfs_text_asset(data: &[u8], patch: &TextAssetPatch) -> Result> { let parser = UnityFsParser::new(); let mut bundle = parser.parse_bytes(data)?; @@ -142,6 +142,13 @@ pub fn patch_unityfs_text_asset(data: &[u8], patch: &TextAssetPatch) -> Result Resu let rebuilt = rebuild_unityfs(&bundle)?; let verified = parser.parse_bytes(&rebuilt)?; + verify_rebuild_preserves_unmodified_content( + &bundle, + &verified, + &patch.serialized_file_path, + patch.path_id, + Some(&patch.field_path), + )?; let serialized = verified .serialized_files .iter() @@ -259,6 +273,13 @@ pub fn patch_unityfs_field(data: &[u8], patch: &FieldPatch) -> Result> { let rebuilt = rebuild_unityfs(&bundle)?; let verified = parser.parse_bytes(&rebuilt)?; + verify_rebuild_preserves_unmodified_content( + &bundle, + &verified, + &patch.serialized_file_path, + patch.path_id, + Some(&patch.field_path), + )?; let serialized = verified .serialized_files .iter() @@ -285,7 +306,12 @@ pub fn patch_unityfs_field(data: &[u8], patch: &FieldPatch) -> Result> { Ok(rebuilt) } -fn rebuild_unityfs(bundle: &UnityFsBundle) -> Result> { +/// Rebuilds a parsed UnityFS bundle while preserving its container shape. +/// +/// The directory order, path, flags, header version and block compression +/// modes are retained. Variable-length file changes are reflected in +/// directory offsets and block sizes; no raw offset is patched blindly. +pub fn rebuild_unityfs_bundle(bundle: &UnityFsBundle) -> Result> { if bundle.files.len() != bundle.directories.len() { return Err(AssetBundleError::Parse(format!( "UnityFS file/directory count mismatch: files={}, directories={}", @@ -293,24 +319,123 @@ fn rebuild_unityfs(bundle: &UnityFsBundle) -> Result> { bundle.directories.len() ))); } - - let mut uncompressed_data = Vec::new(); - let mut directory_offsets = Vec::with_capacity(bundle.files.len()); - for file in &bundle.files { - let offset = u64::try_from(uncompressed_data.len()) - .map_err(|_| AssetBundleError::Parse("UnityFS data offset overflow".to_string()))?; - directory_offsets.push(offset); - uncompressed_data.extend_from_slice(&file.data); + let block_uncompressed_size = bundle + .blocks + .iter() + .try_fold(0u64, |total, block| { + total.checked_add(u64::from(block.uncompressed_size)) + }) + .ok_or_else(|| AssetBundleError::Parse("UnityFS block size overflow".to_string()))?; + let retained_uncompressed_size = u64::try_from(bundle.uncompressed_data.len()) + .map_err(|_| AssetBundleError::Parse("UnityFS data size does not fit u64".to_string()))?; + if block_uncompressed_size != retained_uncompressed_size + || bundle.uncompressed_data_size != retained_uncompressed_size + { + return Err(AssetBundleError::Parse(format!( + "UnityFS block table/data size mismatch: blocks={}, retained={}, declared={}", + block_uncompressed_size, retained_uncompressed_size, bundle.uncompressed_data_size + ))); + } + + let mut ordered_directories = bundle.directories.iter().enumerate().collect::>(); + ordered_directories.sort_by_key(|(_, directory)| directory.offset); + let mut uncompressed_data = Vec::with_capacity(bundle.uncompressed_data.len()); + let mut directory_offsets = vec![0u64; bundle.files.len()]; + let mut cursor = 0usize; + for (index, directory) in ordered_directories { + let file = bundle.files.get(index).ok_or_else(|| { + AssetBundleError::Parse(format!( + "UnityFS directory {} has no corresponding file", + directory.path + )) + })?; + if file.path != directory.path { + return Err(AssetBundleError::Parse(format!( + "UnityFS file/directory path mismatch: file={:?}, directory={:?}", + file.path, directory.path + ))); + } + let start = usize::try_from(directory.offset).map_err(|_| { + AssetBundleError::Parse(format!( + "UnityFS directory {} offset does not fit usize", + directory.path + )) + })?; + let end = start + .checked_add(usize::try_from(directory.size).map_err(|_| { + AssetBundleError::Parse(format!( + "UnityFS directory {} size does not fit usize", + directory.path + )) + })?) + .ok_or_else(|| { + AssetBundleError::Parse(format!( + "UnityFS directory {} range overflows usize", + directory.path + )) + })?; + if start < cursor || end > bundle.uncompressed_data.len() { + return Err(AssetBundleError::Parse(format!( + "UnityFS directory {} overlaps or exceeds the original data region", + directory.path + ))); + } + uncompressed_data.extend_from_slice(&bundle.uncompressed_data[cursor..start]); + directory_offsets[index] = u64::try_from(uncompressed_data.len()).map_err(|_| { + AssetBundleError::Parse("UnityFS directory offset overflow".to_string()) + })?; + uncompressed_data.extend_from_slice(&file.data); + cursor = end; + } + uncompressed_data.extend_from_slice(&bundle.uncompressed_data[cursor..]); + + let block_sizes = repartition_blocks(uncompressed_data.len(), &bundle.blocks)?; + let mut compressed_blocks = Vec::with_capacity(bundle.blocks.len()); + for (index, (block, size)) in bundle.blocks.iter().zip(&block_sizes).enumerate() { + let start = block_sizes[..index].iter().sum::(); + let end = start.checked_add(*size).ok_or_else(|| { + AssetBundleError::Parse(format!("UnityFS rebuilt block {index} range overflows")) + })?; + let bytes = uncompressed_data.get(start..end).ok_or_else(|| { + AssetBundleError::Parse(format!( + "UnityFS rebuilt block {index} range {}..{} exceeds data {}", + start, + end, + uncompressed_data.len() + )) + })?; + compressed_blocks.push(compress_unityfs_bytes( + bytes, + block.compression, + &format!("UnityFS data block {index}"), + )?); } - let data_size = u32::try_from(uncompressed_data.len()).map_err(|_| { - AssetBundleError::Parse("UnityFS rebuilt data exceeds u32 size".to_string()) - })?; let mut blocks_info_body = Vec::new(); - push_i32_be(&mut blocks_info_body, 1); - push_u32_be(&mut blocks_info_body, data_size); - push_u32_be(&mut blocks_info_body, data_size); - push_u16_be(&mut blocks_info_body, 0); + push_i32_be( + &mut blocks_info_body, + i32::try_from(bundle.blocks.len()) + .map_err(|_| AssetBundleError::Parse("UnityFS block count exceeds i32".to_string()))?, + ); + for (block, (bytes, uncompressed_size)) in bundle + .blocks + .iter() + .zip(compressed_blocks.iter().zip(block_sizes.iter())) + { + push_u32_be( + &mut blocks_info_body, + u32::try_from(*uncompressed_size).map_err(|_| { + AssetBundleError::Parse("UnityFS rebuilt block exceeds u32 size".to_string()) + })?, + ); + push_u32_be( + &mut blocks_info_body, + u32::try_from(bytes.len()).map_err(|_| { + AssetBundleError::Parse("UnityFS compressed block exceeds u32 size".to_string()) + })?, + ); + push_u16_be(&mut blocks_info_body, block.flags); + } push_i32_be( &mut blocks_info_body, i32::try_from(bundle.files.len()).map_err(|_| { @@ -328,9 +453,14 @@ fn rebuild_unityfs(bundle: &UnityFsBundle) -> Result> { push_c_string(&mut blocks_info_body, &file.path); } let digest = Md5::digest(&blocks_info_body); - let mut blocks_info = Vec::with_capacity(16 + blocks_info_body.len()); - blocks_info.extend_from_slice(&digest); - blocks_info.extend_from_slice(&blocks_info_body); + let mut blocks_info_uncompressed = Vec::with_capacity(16 + blocks_info_body.len()); + blocks_info_uncompressed.extend_from_slice(&digest); + blocks_info_uncompressed.extend_from_slice(&blocks_info_body); + let blocks_info = compress_unityfs_bytes( + &blocks_info_uncompressed, + compression_from_header_flags(bundle.header.flags), + "UnityFS block info", + )?; let mut output = Vec::new(); push_c_string(&mut output, "UnityFS"); @@ -342,27 +472,395 @@ fn rebuild_unityfs(bundle: &UnityFsBundle) -> Result> { push_u32_be( &mut output, u32::try_from(blocks_info.len()).map_err(|_| { - AssetBundleError::Parse("UnityFS block info exceeds u32 size".to_string()) + AssetBundleError::Parse("UnityFS compressed block info exceeds u32 size".to_string()) })?, ); push_u32_be( &mut output, - u32::try_from(blocks_info.len()).map_err(|_| { + u32::try_from(blocks_info_uncompressed.len()).map_err(|_| { AssetBundleError::Parse("UnityFS block info exceeds u32 size".to_string()) })?, ); - push_u32_be(&mut output, 0); + push_u32_be(&mut output, bundle.header.flags); if bundle.header.format_version >= 7 { align_vec(&mut output, 16); } - output.extend_from_slice(&blocks_info); - output.extend_from_slice(&uncompressed_data); + if block_info_at_end(bundle.header.flags) { + if block_data_aligned(bundle.header.flags) { + align_vec(&mut output, 16); + } + for bytes in &compressed_blocks { + output.extend_from_slice(bytes); + } + output.extend_from_slice(&blocks_info); + } else { + output.extend_from_slice(&blocks_info); + if block_data_aligned(bundle.header.flags) { + align_vec(&mut output, 16); + } + for bytes in &compressed_blocks { + output.extend_from_slice(bytes); + } + } let total_size = u64::try_from(output.len()) .map_err(|_| AssetBundleError::Parse("UnityFS rebuilt size overflow".to_string()))?; output[total_size_offset..total_size_offset + 8].copy_from_slice(&total_size.to_be_bytes()); Ok(output) } +fn rebuild_unityfs(bundle: &UnityFsBundle) -> Result> { + rebuild_unityfs_bundle(bundle) +} + +fn compression_from_header_flags(flags: u32) -> UnityFsCompression { + crate::parser::compression_from_flags((flags & 0x3f) as u16) +} + +fn block_info_at_end(flags: u32) -> bool { + flags & 0x80 != 0 +} + +fn block_data_aligned(flags: u32) -> bool { + flags & 0x200 != 0 +} + +fn repartition_blocks( + total_size: usize, + blocks: &[crate::types::UnityFsBlockInfo], +) -> Result> { + if blocks.is_empty() { + return Err(AssetBundleError::Parse( + "UnityFS bundle has no blocks to rebuild".to_string(), + )); + } + let original_total = blocks + .iter() + .try_fold(0u64, |total, block| { + total.checked_add(u64::from(block.uncompressed_size)) + }) + .ok_or_else(|| AssetBundleError::Parse("UnityFS block size overflow".to_string()))?; + if original_total == 0 { + return Ok(vec![0; blocks.len()]); + } + let total = u64::try_from(total_size) + .map_err(|_| AssetBundleError::Parse("UnityFS data size does not fit u64".to_string()))?; + let mut result = Vec::with_capacity(blocks.len()); + let mut previous = 0u64; + let mut cumulative = 0u64; + for block in blocks { + cumulative = cumulative + .checked_add(u64::from(block.uncompressed_size)) + .ok_or_else(|| { + AssetBundleError::Parse("UnityFS block boundary overflow".to_string()) + })?; + let boundary = total.checked_mul(cumulative).ok_or_else(|| { + AssetBundleError::Parse("UnityFS block boundary overflow".to_string()) + })? / original_total; + result.push(usize::try_from(boundary - previous).map_err(|_| { + AssetBundleError::Parse("UnityFS rebuilt block size does not fit usize".to_string()) + })?); + previous = boundary; + } + Ok(result) +} + +fn compress_unityfs_bytes( + data: &[u8], + compression: UnityFsCompression, + context: &str, +) -> Result> { + match compression { + UnityFsCompression::None => Ok(data.to_vec()), + UnityFsCompression::Lz4 => { + lz4::block::compress(data, Some(lz4::block::CompressionMode::FAST(1)), false).map_err( + |error| { + AssetBundleError::Parse(format!("Failed to compress {context} as LZ4: {error}")) + }, + ) + } + UnityFsCompression::Lz4Hc => lz4::block::compress( + data, + Some(lz4::block::CompressionMode::HIGHCOMPRESSION(9)), + false, + ) + .map_err(|error| { + AssetBundleError::Parse(format!("Failed to compress {context} as LZ4HC: {error}")) + }), + UnityFsCompression::Lzma => { + let mut output = Vec::new(); + lzma_rs::lzma_compress(&mut std::io::Cursor::new(data), &mut output).map_err( + |error| { + AssetBundleError::Parse(format!( + "Failed to compress {context} as LZMA: {error}" + )) + }, + )?; + Ok(output) + } + UnityFsCompression::Unknown(value) => Err(AssetBundleError::UnsupportedFormat(format!( + "unsupported {context} compression flag: {value}" + ))), + } +} + +fn verify_rebuild_preserves_unmodified_content( + original: &UnityFsBundle, + rebuilt_bytes: &UnityFsBundle, + modified_serialized_path: &str, + modified_path_id: i64, + modified_field_path: Option<&str>, +) -> Result<()> { + if original.header.format_version != rebuilt_bytes.header.format_version + || original.header.target_version != rebuilt_bytes.header.target_version + || original.header.unity_version != rebuilt_bytes.header.unity_version + || original.header.flags != rebuilt_bytes.header.flags + { + return Err(AssetBundleError::Parse( + "UnityFS rebuild changed immutable header metadata".to_string(), + )); + } + if original.blocks.len() != rebuilt_bytes.blocks.len() + || original.directories.len() != rebuilt_bytes.directories.len() + { + return Err(AssetBundleError::Parse( + "UnityFS rebuild changed block or directory count".to_string(), + )); + } + if original.serialized_files.len() != rebuilt_bytes.serialized_files.len() { + return Err(AssetBundleError::Parse( + "UnityFS rebuild changed serialized-file count".to_string(), + )); + } + if original.serialized_parse_errors != rebuilt_bytes.serialized_parse_errors { + return Err(AssetBundleError::Parse( + "UnityFS rebuild changed serialized-file diagnostics".to_string(), + )); + } + for (original, rebuilt) in original.blocks.iter().zip(&rebuilt_bytes.blocks) { + if original.flags != rebuilt.flags || original.compression != rebuilt.compression { + return Err(AssetBundleError::Parse( + "UnityFS rebuild changed block compression metadata".to_string(), + )); + } + } + for (original, rebuilt) in original.directories.iter().zip(&rebuilt_bytes.directories) { + if original.path != rebuilt.path || original.flags != rebuilt.flags { + return Err(AssetBundleError::Parse( + "UnityFS rebuild changed directory path or flags".to_string(), + )); + } + } + + for (original_file, rebuilt_file) in original.files.iter().zip(&rebuilt_bytes.files) { + if original_file.path != modified_serialized_path + && (original_file.path != rebuilt_file.path || original_file.data != rebuilt_file.data) + { + return Err(AssetBundleError::Parse(format!( + "UnityFS rebuild changed an unmodified directory file: {}", + original_file.path + ))); + } + } + + for original_file in &original.serialized_files { + let rebuilt_file = rebuilt_bytes + .serialized_files + .iter() + .find(|candidate| candidate.source_path == original_file.source_path) + .ok_or_else(|| { + AssetBundleError::Parse(format!( + "UnityFS rebuild lost serialized file {:?}", + original_file.source_path + )) + })?; + if original_file.source_path != rebuilt_file.source_path + || original_file.version != rebuilt_file.version + || original_file.unity_version != rebuilt_file.unity_version + || original_file.platform != rebuilt_file.platform + || original_file.types != rebuilt_file.types + || original_file.objects.len() != rebuilt_file.objects.len() + { + return Err(AssetBundleError::Parse( + "UnityFS rebuild changed serialized-file metadata".to_string(), + )); + } + for (original_object, rebuilt_object) in + original_file.objects.iter().zip(&rebuilt_file.objects) + { + if original_object.path_id != rebuilt_object.path_id + || original_object.type_index != rebuilt_object.type_index + || original_object.class_id != rebuilt_object.class_id + { + return Err(AssetBundleError::Parse(format!( + "UnityFS rebuild changed object table entry path_id {}", + original_object.path_id + ))); + } + let is_modified_object = original_file.source_path.as_deref() + == Some(modified_serialized_path) + && original_object.path_id == modified_path_id; + if !is_modified_object && original_object.byte_size != rebuilt_object.byte_size { + return Err(AssetBundleError::Parse(format!( + "UnityFS rebuild changed unmodified object byte size path_id {}", + original_object.path_id + ))); + } + if is_modified_object { + if let Some(field_path) = modified_field_path { + let original_fields = original_file.fields_for_object_entry(original_object)?; + let rebuilt_fields = rebuilt_file.fields_for_object_entry(rebuilt_object)?; + verify_unmodified_fields( + &original_fields, + &rebuilt_fields, + field_path, + original_object.path_id, + )?; + } else { + let original_asset = original_file + .text_assets() + .iter() + .find(|asset| asset.path_id == modified_path_id); + let rebuilt_asset = rebuilt_file + .text_assets() + .iter() + .find(|asset| asset.path_id == modified_path_id); + if original_asset.map(|asset| &asset.name) + != rebuilt_asset.map(|asset| &asset.name) + { + return Err(AssetBundleError::Parse(format!( + "TextAsset path_id {} name changed while rebuilding", + modified_path_id + ))); + } + } + } else { + if original_file.object_bytes(original_object)? + != rebuilt_file.object_bytes(rebuilt_object)? + { + return Err(AssetBundleError::Parse(format!( + "UnityFS rebuild changed unmodified object path_id {}", + original_object.path_id + ))); + } + } + } + } + Ok(()) +} + +fn verify_unmodified_fields( + original: &[UnitySerializedField], + rebuilt: &[UnitySerializedField], + modified_field_path: &str, + path_id: i64, +) -> Result<()> { + if original.len() != rebuilt.len() { + return Err(AssetBundleError::Parse(format!( + "Unity object path_id {path_id} changed field count while rebuilding" + ))); + } + for (original, rebuilt) in original.iter().zip(rebuilt) { + if original.path != rebuilt.path + || original.name != rebuilt.name + || original.type_name != rebuilt.type_name + || original.type_tree_node_index != rebuilt.type_tree_node_index + { + return Err(AssetBundleError::Parse(format!( + "Unity object path_id {path_id} changed field metadata at {}", + original.path + ))); + } + if !field_path_contains(&original.path, modified_field_path) + && original.byte_size != rebuilt.byte_size + { + return Err(AssetBundleError::Parse(format!( + "Unity object path_id {path_id} changed unmodified field byte size at {}", + original.path + ))); + } + if original.path == modified_field_path { + continue; + } + verify_unmodified_values( + &original.value, + &rebuilt.value, + modified_field_path, + path_id, + )?; + } + Ok(()) +} + +fn field_path_contains(parent: &str, child: &str) -> bool { + child == parent + || child + .strip_prefix(parent) + .is_some_and(|suffix| suffix.starts_with('.') || suffix.starts_with('[')) +} + +fn verify_unmodified_values( + original: &UnitySerializedValue, + rebuilt: &UnitySerializedValue, + modified_field_path: &str, + path_id: i64, +) -> Result<()> { + match (original, rebuilt) { + (UnitySerializedValue::Object(original), UnitySerializedValue::Object(rebuilt)) + | (UnitySerializedValue::Array(original), UnitySerializedValue::Array(rebuilt)) + | (UnitySerializedValue::Map(original), UnitySerializedValue::Map(rebuilt)) => { + verify_unmodified_fields(original, rebuilt, modified_field_path, path_id) + } + ( + UnitySerializedValue::ManagedReference { + type_name: original_type, + metadata: original_metadata, + fields: original_fields, + bytes: original_bytes, + }, + UnitySerializedValue::ManagedReference { + type_name: rebuilt_type, + metadata: rebuilt_metadata, + fields: rebuilt_fields, + bytes: rebuilt_bytes, + }, + ) => { + if original_type != rebuilt_type + || original_metadata != rebuilt_metadata + || original_bytes != rebuilt_bytes + { + return Err(AssetBundleError::Parse(format!( + "Unity object path_id {path_id} changed managed-reference metadata" + ))); + } + verify_unmodified_fields( + original_fields, + rebuilt_fields, + modified_field_path, + path_id, + ) + } + ( + UnitySerializedValue::ManagedReferenceRegistry { + references: _, + fields: original_fields, + }, + UnitySerializedValue::ManagedReferenceRegistry { + references: _, + fields: rebuilt_fields, + }, + ) => verify_unmodified_fields( + original_fields, + rebuilt_fields, + modified_field_path, + path_id, + ), + (original, rebuilt) if original != rebuilt => Err(AssetBundleError::Parse(format!( + "Unity object path_id {path_id} changed an unmodified field value" + ))), + _ => Ok(()), + } +} + fn find_field_value<'a>( fields: &'a [UnitySerializedField], field_path: &str, @@ -489,6 +987,99 @@ mod tests { data } + fn synthetic_bundle_with_compressed_blocks( + files: &[(&str, &[u8])], + block_flags: u16, + header_flags: u32, + ) -> Vec { + let mut uncompressed = Vec::new(); + let mut directory_offsets = Vec::with_capacity(files.len()); + for (_, bytes) in files { + directory_offsets.push(uncompressed.len() as u64); + uncompressed.extend_from_slice(bytes); + } + let split = uncompressed.len() / 2; + let split = if uncompressed.is_empty() { + 0 + } else { + split.max(1).min(uncompressed.len()) + }; + let chunks = [&uncompressed[..split], &uncompressed[split..]]; + let compressed_chunks = chunks + .iter() + .map(|chunk| match block_flags & 0x3f { + 0 => chunk.to_vec(), + 1 => { + let mut output = Vec::new(); + lzma_rs::lzma_compress(&mut std::io::Cursor::new(chunk), &mut output).unwrap(); + output + } + 2..=4 => lz4::block::compress(chunk, None, false).unwrap(), + value => panic!("unsupported fixture compression {value}"), + }) + .collect::>(); + + let mut block_info_body = Vec::new(); + push_i32_be(&mut block_info_body, chunks.len() as i32); + for (chunk, compressed) in chunks.iter().zip(&compressed_chunks) { + push_u32_be(&mut block_info_body, chunk.len() as u32); + push_u32_be(&mut block_info_body, compressed.len() as u32); + push_u16_be(&mut block_info_body, block_flags); + } + push_i32_be(&mut block_info_body, files.len() as i32); + for ((path, bytes), offset) in files.iter().zip(directory_offsets) { + push_u64_be(&mut block_info_body, offset); + push_u64_be(&mut block_info_body, bytes.len() as u64); + push_u32_be(&mut block_info_body, 0); + push_c_string(&mut block_info_body, path); + } + let mut block_info = vec![0; 16]; + block_info.extend_from_slice(&block_info_body); + let block_info = match header_flags & 0x3f { + 0 => block_info, + 1 => { + let mut output = Vec::new(); + lzma_rs::lzma_compress(&mut std::io::Cursor::new(&block_info), &mut output) + .unwrap(); + output + } + 2..=4 => lz4::block::compress(&block_info, None, false).unwrap(), + value => panic!("unsupported fixture block-info compression {value}"), + }; + + let mut data = Vec::new(); + push_c_string(&mut data, "UnityFS"); + push_u32_be(&mut data, 8); + push_c_string(&mut data, "5.x.x"); + push_c_string(&mut data, "2021.3.56f2"); + let total_size_offset = data.len(); + push_u64_be(&mut data, 0); + push_u32_be(&mut data, block_info.len() as u32); + push_u32_be(&mut data, (16 + block_info_body.len()) as u32); + push_u32_be(&mut data, header_flags); + align_vec(&mut data, 16); + if header_flags & 0x80 != 0 { + if header_flags & 0x200 != 0 { + align_vec(&mut data, 16); + } + for compressed in &compressed_chunks { + data.extend_from_slice(compressed); + } + data.extend_from_slice(&block_info); + } else { + data.extend_from_slice(&block_info); + if header_flags & 0x200 != 0 { + align_vec(&mut data, 16); + } + for compressed in &compressed_chunks { + data.extend_from_slice(compressed); + } + } + let total_size = data.len() as u64; + data[total_size_offset..total_size_offset + 8].copy_from_slice(&total_size.to_be_bytes()); + data + } + fn push_i16_le(data: &mut Vec, value: i16) { data.extend_from_slice(&value.to_le_bytes()); } @@ -549,6 +1140,66 @@ mod tests { file } + fn synthetic_serialized_two_text_assets(first: &[u8], second: &[u8]) -> Vec { + fn text_asset_object(bytes: &[u8]) -> Vec { + let mut object = Vec::new(); + push_u32_le(&mut object, 8); + object.extend_from_slice(b"Scenario"); + align_vec(&mut object, 4); + push_u32_le(&mut object, bytes.len() as u32); + object.extend_from_slice(bytes); + object + } + + let first_object = text_asset_object(first); + let second_object = text_asset_object(second); + let mut object_data = Vec::new(); + let first_offset = object_data.len() as u64; + object_data.extend_from_slice(&first_object); + let first_size = object_data.len() as u64 - first_offset; + let second_offset = object_data.len() as u64; + object_data.extend_from_slice(&second_object); + let second_size = object_data.len() as u64 - second_offset; + + let mut metadata = Vec::new(); + metadata.extend_from_slice(b"2021.3.56f2\0"); + push_i32_le(&mut metadata, 19); + metadata.push(0); + push_i32_le(&mut metadata, 1); + push_i32_le(&mut metadata, 49); + metadata.push(0); + push_i16_le(&mut metadata, 0); + metadata.extend_from_slice(&[0; 16]); + push_i32_le(&mut metadata, 2); + align_vec(&mut metadata, 4); + for (path_id, offset, size) in [ + (1i64, first_offset, first_size), + (2, second_offset, second_size), + ] { + metadata.extend_from_slice(&path_id.to_le_bytes()); + push_u64_le(&mut metadata, offset); + push_u32_le(&mut metadata, size as u32); + push_i32_le(&mut metadata, 0); + } + + let header_len = 48usize; + let data_offset = header_len + metadata.len(); + let file_size = data_offset + object_data.len(); + let mut file = Vec::new(); + push_u32_be(&mut file, metadata.len() as u32); + push_u32_be(&mut file, file_size as u32); + push_u32_be(&mut file, 22); + push_u32_be(&mut file, 0); + file.extend_from_slice(&[0, 0, 0, 0]); + push_u32_be(&mut file, metadata.len() as u32); + push_u64_be(&mut file, file_size as u64); + push_u64_be(&mut file, data_offset as u64); + push_u64_be(&mut file, 0); + file.extend_from_slice(&metadata); + file.extend_from_slice(&object_data); + file + } + fn synthetic_serialized_monobehaviour() -> Vec { let mut object_data = Vec::new(); push_u32_le(&mut object_data, 5); @@ -1391,6 +2042,135 @@ mod tests { assert_eq!(parsed.unity_version, reparsed.unity_version); } + #[test] + fn rebuild_preserves_compressed_blocks_alignment_and_untouched_files() { + let serialized = synthetic_serialized_text_asset(b"old"); + let source = synthetic_bundle_with_compressed_blocks( + &[("CAB-scenario", &serialized), ("CAB-untouched", b"keep")], + 2, + 2 | 0x200, + ); + let parsed = UnityFsParser::new().parse_bytes(&source).unwrap(); + assert_eq!(parsed.blocks.len(), 2); + assert_eq!(parsed.blocks[0].compression, UnityFsCompression::Lz4); + assert_eq!(parsed.header.flags, 2 | 0x200); + + let patched = patch_unityfs_text_asset( + &source, + &TextAssetPatch { + serialized_file_path: "CAB-scenario".to_string(), + path_id: 1, + expected_name: Some("Scenario".to_string()), + replacement: b"a longer localized payload".to_vec(), + }, + ) + .unwrap(); + let reparsed = UnityFsParser::new().parse_bytes(&patched).unwrap(); + + assert_eq!(reparsed.blocks.len(), 2); + assert_eq!(reparsed.blocks[0].compression, UnityFsCompression::Lz4); + assert_eq!(reparsed.blocks[1].compression, UnityFsCompression::Lz4); + assert_eq!(reparsed.header.flags, parsed.header.flags); + assert_eq!( + reparsed + .files + .iter() + .find(|file| file.path == "CAB-untouched") + .unwrap() + .data, + b"keep" + ); + assert_eq!(reparsed.text_assets[0].bytes, b"a longer localized payload"); + } + + #[test] + fn rebuild_preserves_unmodified_serialized_objects() { + let serialized = synthetic_serialized_two_text_assets(b"old", b"keep-object"); + let source = synthetic_bundle_with_path(&serialized, "CAB-scenario"); + let patched = patch_unityfs_text_asset( + &source, + &TextAssetPatch { + serialized_file_path: "CAB-scenario".to_string(), + path_id: 1, + expected_name: Some("Scenario".to_string()), + replacement: b"localized".to_vec(), + }, + ) + .unwrap(); + let reparsed = UnityFsParser::new().parse_bytes(&patched).unwrap(); + + assert_eq!( + reparsed + .text_assets + .iter() + .find(|asset| asset.path_id == 1) + .unwrap() + .bytes, + b"localized" + ); + assert_eq!( + reparsed + .text_assets + .iter() + .find(|asset| asset.path_id == 2) + .unwrap() + .bytes, + b"keep-object" + ); + } + + #[test] + fn rebuild_preserves_block_info_at_end_and_lzma_compression() { + let serialized = synthetic_serialized_text_asset(b"old"); + let source = synthetic_bundle_with_compressed_blocks( + &[("CAB-scenario", &serialized)], + 1, + 1 | 0x80 | 0x200, + ); + let patched = patch_unityfs_text_asset( + &source, + &TextAssetPatch { + serialized_file_path: "CAB-scenario".to_string(), + path_id: 1, + expected_name: None, + replacement: b"localized".to_vec(), + }, + ) + .unwrap(); + let reparsed = UnityFsParser::new().parse_bytes(&patched).unwrap(); + + assert_eq!(reparsed.header.flags, 1 | 0x80 | 0x200); + assert_eq!(reparsed.blocks[0].compression, UnityFsCompression::Lzma); + assert_eq!(reparsed.blocks[1].compression, UnityFsCompression::Lzma); + assert_eq!(reparsed.text_assets[0].bytes, b"localized"); + } + + #[test] + fn rebuild_rejects_unknown_compression_without_guessing() { + let source = synthetic_bundle(b"payload"); + let mut parsed = UnityFsParser::new().parse_bytes(&source).unwrap(); + parsed.blocks[0].compression = UnityFsCompression::Unknown(63); + + let error = rebuild_unityfs_bundle(&parsed).unwrap_err(); + + assert!(matches!( + error, + AssetBundleError::UnsupportedFormat(message) + if message.contains("unsupported UnityFS data block 0 compression flag: 63") + )); + } + + #[test] + fn rebuild_rejects_inconsistent_uncompressed_block_table() { + let source = synthetic_bundle(b"payload"); + let mut parsed = UnityFsParser::new().parse_bytes(&source).unwrap(); + parsed.blocks[0].uncompressed_size += 1; + + let error = rebuild_unityfs_bundle(&parsed).unwrap_err().to_string(); + + assert!(error.contains("block table/data size mismatch"), "{error}"); + } + #[test] fn patches_text_asset_and_verifies_reparsed_payload() { let original_text = "こんにちは".as_bytes(); diff --git a/crates/bat-assetbundle/src/serialized.rs b/crates/bat-assetbundle/src/serialized.rs index c620968..6869c0c 100644 --- a/crates/bat-assetbundle/src/serialized.rs +++ b/crates/bat-assetbundle/src/serialized.rs @@ -1088,6 +1088,34 @@ impl UnitySerializedFile { } } + /// Returns the raw payload bytes for one object table entry. + /// + /// The returned slice is still owned by the parsed serialized file. It is + /// useful to compare untouched objects across a variable-length rebuild. + pub fn object_bytes(&self, object: &UnitySerializedObject) -> Result<&[u8]> { + let object_start = self + .data_offset + .checked_add(usize::try_from(object.byte_start).map_err(|_| { + AssetBundleError::Parse(format!( + "Unity object path_id {} byte_start does not fit usize", + object.path_id + )) + })?) + .ok_or_else(|| AssetBundleError::Parse("Unity object offset overflow".to_string()))?; + let object_end = object_start + .checked_add(object.byte_size as usize) + .ok_or_else(|| AssetBundleError::Parse("Unity object size overflow".to_string()))?; + self.raw_data.get(object_start..object_end).ok_or_else(|| { + AssetBundleError::Parse(format!( + "Unity object path_id {} byte range {}..{} exceeds file size {}", + object.path_id, + object_start, + object_end, + self.raw_data.len() + )) + }) + } + fn rewrite_object_payload( &self, target_index: usize, diff --git a/crates/bat-assetbundle/src/types.rs b/crates/bat-assetbundle/src/types.rs index e49c955..102ad42 100644 --- a/crates/bat-assetbundle/src/types.rs +++ b/crates/bat-assetbundle/src/types.rs @@ -62,6 +62,9 @@ pub struct UnityFsBundle { pub uncompressed_data_size: u64, /// Original bytes retained for future extraction/serialization. pub raw_data: Vec, + /// Uncompressed data region retained so rebuilds can preserve gaps and + /// trailing bytes that are not represented by directory entries. + pub(crate) uncompressed_data: Vec, /// Files extracted from the UnityFS uncompressed data region. pub files: Vec, /// Serialized files parsed from UnityFS directory files. diff --git a/docs/architecture/assetbundle.md b/docs/architecture/assetbundle.md index 2456d63..8e15701 100644 --- a/docs/architecture/assetbundle.md +++ b/docs/architecture/assetbundle.md @@ -29,10 +29,10 @@ | --- | --- | --- | --- | | 官方 seed manifest | `TableCatalog.bytes`、`BundlePackingInfo.bytes`、`MediaCatalog.bytes` | 完整下载 URL、相对路径、hash 校验边界 | 已用于下载计划,仍需沉淀更多结构化字段 | | Addressables catalog | `catalog_*.zip` 内 JSON/bin catalog、`catalog_*.hash` | asset path、provider、dependencies、size、CRC、bundle name | JSON/compact 当前目标字段已覆盖;未知结构返回明确错误 | -| UnityFS container | `.bundle`、zip 内 bundle | header、block、directory、解压文件、基础摘要 | 已支持基础解包、LZ4/LZMA、alignment、大小/计数/路径/边界校验 | +| UnityFS container | `.bundle`、zip 内 bundle | header、block、directory、解压文件、基础摘要 | 已支持基础解包、LZ4/LZMA、alignment、大小/计数/路径/边界校验;受支持 patch 可保留容器形态重建 | | Serialized file | UnityFS directory 文件 | header、type table、TypeTree node、object table、TextAsset bytes | 已支持基础表结构和 TextAsset bytes | | Unity 对象字段 | TextAsset、MonoBehaviour、ScriptableObject | 可翻译文本单元、上下文、资源定位 | TypeTree 基础字段读取、`SerializedReference` / prefixed managed-reference metadata alias、payload 提取和字符串提取已落地,真实结构覆盖继续扩大 | -| Patch 发布 | 已翻译 TextUnit、中间格式、原版资源 | 可验证 localized patch manifest、汉化 release 目录、current/state | TextAsset、TypeTree string field 和 managed-reference string field 的 localized publish/rollback 已落地;整体 AssetBundle 重打包与通用 manifest 发布仍未完成 | +| Patch 发布 | 已翻译 TextUnit、中间格式、原版资源 | 可验证 localized patch manifest、汉化 release 目录、current/state | 已支持 TextAsset、TypeTree string field、managed-reference string field 的 UnityFS parse→modify→rebuild→reparse;可验证 ZIP 内 bundle 时会重写外层 ZIP;通用 manifest 发布仍未完成 | --- @@ -51,13 +51,15 @@ 9. `TextUnitExtractor` 已把 JSON/CSV/TSV/plain TextAsset、TypeTree 字符串字段和 TypeTree-covered managed reference payload 字符串输出为可序列化 TextUnit/JSONL;zip 场景保留 archive entry,TextUnit 明细包含 serialized file、path id、class id、field path、字段 offset/byte size、format、asset name 和上下文。managed-reference 类型元数据保留为 payload context,不进入翻译文本队列;即使 registry 暂时只能走 fallback 字段遍历,`RefIds`、`className`、`namespaceName`、`asmName` 等元数据别名也会被跳过,payload/value/object 家族和 `managedReferenceData` / `referenceData` / `serializedData` 仍按 payload 处理,并按 `RefIds[n]` 等记录前缀或子字段推导 metadata,避免多条 fallback record 混用 managed-reference context。 10. `ResourceImportService` 能把 AssetBundle 摘要、TextAsset/Table/Media 分类和 TextUnit 摘要写入导入报告。 11. 官方同步后 `OfficialParseCacheService` 能从 `official-download-manifest.json` 遍历所有资源,解析直接 bundle 和 zip 内条目,非候选资源记录为 unsupported,并缓存 TextUnit 数量/格式/诊断摘要,同时写出 `official-textunit-index.json` 供 `parse.text_units` / `parse.errors` 查询。 +12. `rebuild_unityfs_bundle` 保留当前已验证 bundle 的 header 版本、directory 顺序/路径/flags、block 数量/压缩模式、block-info-at-end 和 block-data alignment;变长目录文件会重新计算目录 offset、block size、block-info hash 和总大小,未知压缩模式明确拒绝。 +13. 重建后会重解析并校验未修改 directory 文件、serialized object table、未修改 object raw bytes,以及目标对象的未修改 TypeTree 字段;不能证明保真的结构不会发布。 当前还不能宣称完整: 1. TypeTree-covered managed reference 字段和 registry 记录已可结构化解码并参与文本提取,常见 registry 命名别名(含 `m_ManagedReferences`、`RefIds`、`m_RefIds`、verbose type 字段)、metadata 命名别名(含 `id`、`typeInfo`)、payload 命名别名(含 `data`、`value`、`payload`、`object`、`managedReferencePayload`、`referencePayload`、`serializedReferencePayload`、`managedReferenceValue`、`referenceValue`、`serializedReferenceValue`、`managedReferenceObject`、`referenceObject`、`serializedReferenceObject`、`managedReferenceData`、`referenceData`、`serializedData`)、full typename 拆解和 payload-only TextUnit 提取已有回归覆盖,多记录 registry 聚合也已有单元回归;fallback 字段遍历会跳过常见 registry 元数据字符串,避免误入翻译队列,并按记录前缀或子字段可推导 metadata 保留 managed-reference TextUnit context。enum `value__` backing field 和 `LayerMask` / `BitField` 的 `m_Bits` backing field 已可语义化解码和替换;`Vector2f/3f/4f`、`Quaternionf`、`ColorRGBA`、`Rectf`、`AABB/Bounds/Ray`、`Matrix4x4f`、`Vector2Int/Vector3Int`、`RectInt`、`BoundsInt`、`RangeInt`、`GUID`、`Hash128` 等固定 Unity 值类型的 leaf 和 direct child TypeTree 形态已可结构化解码和语义替换;array/vector/staticvector/List/HashSet/map 元素与 registry payload 字段已保留独立 field path、offset 和 byte size,可用于字符串元素 patch,managed-reference registry payload 字符串、enum、bit_field、unknown fixed-size raw bytes、object 字段组合和 TypeTree schema 支撑的 array/vector/List/HashSet/map 已可整体变长替换,`first/second` 与 `key/value` map entry schema 已有 serialized 和 UnityFS 重建回归,ScriptableObject `key/value` map 解析、变长替换和 UnityFS 重建已有专门回归,且嵌套 vector `Array`、`List` / `HashSet` 集合 alias、enum、bit_field、unknown fixed-size raw bytes 与 managed-reference payload 字段已有重建回归覆盖;后续仍需继续补齐真实样本驱动的完整 managed reference registry / map entry 变体、unknown 字段结构语义和版本差异。 2. Addressables 当前目标 JSON/compact 字段链已补齐;未识别的独立二进制格式仍返回明确错误,不静默降级。 3. 官方 release 已可配置导入 CAS + ResourceRepository,并可通过 `resource.index` 查询现有资源索引;Resource metadata 已记录 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 摘要。 -4. 不能完成复杂对象字段重打包,也不能从真实 Crowdin 结果自动生成完整汉化文件集合。 +4. 尚未覆盖所有真实 Unity 版本、未知字段语义和任意复杂 AssetBundle 结构,也不能从真实 Crowdin 结果自动生成完整汉化文件集合;当前仅对已有真实/合成回归覆盖的结构宣称支持。 --- @@ -170,10 +172,10 @@ catalog 仍按“明确不支持”处理,不把低保真路径伪装成完整 交付: 1. 已定义 `localized-patch-manifest.json`:目标官方版本、localized release、输出文件、hash、size、byte delta、TextUnit/provider/review trace 和回滚信息。 -2. 已支持 UnityFS TextAsset、TypeTree string field 和 managed-reference string field 的 localized patch 操作。 +2. 已支持 UnityFS TextAsset、TypeTree string field 和 managed-reference string field 的 localized patch 操作;普通 ZIP 条目在 `archive_entry` 可验证、内层可重解析时会解包、重建并重写外层 ZIP,路径穿越、symlink、混合直接/ZIP patch 和无效内层 bundle 明确失败。 3. MonoBehaviour/ScriptableObject 字段替换必须依赖 P2 字段级解析结果。 4. Patch 产物写入配置化汉化发布根下的 `.staging/`,校验通过后发布到 `versions/` 并切换 `current`;rollback 按 manifest 恢复上一 release。 -5. 成功后发布状态从 `not_localized` 切到 `localized`;`localized.status` 要求 state、current symlink 和 patch manifest 同时匹配当前官方 release。 +5. 成功后发布状态从 `not_localized` 切到 `localized`;`localized.status` 要求 state、current symlink 和 patch manifest 同时匹配当前官方 release,ZIP 外层文件和内层 UnityFS 也必须通过发布后重解析校验。 验收: diff --git a/docs/architecture/official-resource-backend.md b/docs/architecture/official-resource-backend.md index f8e7dc9..dddd1f1 100644 --- a/docs/architecture/official-resource-backend.md +++ b/docs/architecture/official-resource-backend.md @@ -244,7 +244,7 @@ trusted 和 release/TextUnit/provider/run provenance;`translation.tasks` 优 13. 发布完成后先对比上一完整 release 和当前 release 的 `official-download-manifest.json`,写出 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`。同一 destination 只有 size 或 BLAKE3 变化才算 modified;新增+变更资源进入解析/翻译 handoff,删除资源只进入差异记录。当前只预留 Crowdin 本地 handoff,不发外部 API 请求。 14. 随后刷新 active release 下的 `official-parse-cache.json` 和 `official-textunit-index.json`,并从 Added/Modified 资源、parse cache 与 TextUnit 明细索引派生 `official-textunit-tasks.json`、`crowdin-textunit-queue.json` 和版本化的 `translation-tasks.sqlite`;up-to-date 轮询在已有有效解析缓存、TextUnit 明细索引和 TextUnit 队列时只读取摘要,不重复解析,重新同步队列时保留已有 worker 状态。 15. 若启用 `--import-repository`,已校验 release 会被导入 CAS + `ResourceRepository`,并可经 `resource.index` 查询。历史 release 候选失效时,已有 CAS 对象会先经过完整性和元数据校验,再增加 release 引用并原子物化;当前 release 在 `official-cas-reuse-references.json` 中记录引用,staging/release 清理时递减,失败则回退网络并保留诊断。 -16. 官方同步报告默认给出 `localized_release_status=not_localized`,表示原版资源已发布、汉化资源未发布;UnityFS TextAsset patch 发布成功并通过 `localized-patch-manifest.json`、current symlink 和 release ID 校验后,`localized.status` 才返回 `localized`,表示原版和汉化两套资源都已发布。`translation.proofread` 只会把 workflow 标记成 `manual_proofreading` / `translation.manual_proofreading`,不会回退已发布汉化 release 的发布状态。 +16. 官方同步报告默认给出 `localized_release_status=not_localized`,表示原版资源已发布、汉化资源未发布;受支持的 UnityFS TextAsset、TypeTree string field 和 managed-reference string field patch 发布成功并通过 `localized-patch-manifest.json`、current symlink、release ID 及 ZIP 内层重解析校验后,`localized.status` 才返回 `localized`,表示原版和汉化两套资源都已发布。`translation.proofread` 只会把 workflow 标记成 `manual_proofreading` / `translation.manual_proofreading`,不会回退已发布汉化 release 的发布状态。 维护期特殊分支:如果官方 launcher/server-info 已经指向新资源根,但 client-patch seed marker 或必需 seed catalog 仍返回 403/404 等未开放状态,`bat` 返回 `waiting_for_official_resources`,保留现有 `current`,不创建失败 staging;若本轮启用 `--auto-discover`,会在 `/official-launcher-bootstrap.pending.json` 写入待处理 launcher bootstrap 证据,供后续排障和自研客户端开发使用。 @@ -347,7 +347,8 @@ JSON-RPC 2.0 服务,是面向上层服务(Go 层)的**主要跨语言边 `resource.state/sync/verify/repair/manifest/list/index`、`parse.status/text_units/errors`、 `translation.tasks/handoff/task.update/proofread`、`localized.status`、`catalog.*` 与 `task.status/list/cancel/logs` 已实现;文件级 `patch.apply` / `unityfs.patch_*` - 已实现,发布级 patch 与复杂 UnityFS 语义编辑待引擎; + 和受支持 localized publish/rollback 已实现,`archive_entry` 可验证时会重写 + 外层 ZIP;通用发布级 patch 与复杂 UnityFS 语义编辑仍待后续; `task.create` 按设计暂不开放通用任务入口; `daemon.restart` 通过 Rust lifecycle controller 复用 CLI restart 路径; `daemon.clean-stable` 仍由 CLI 侧按进程生命周期显式执行。 diff --git a/docs/guides/bat-workflows.md b/docs/guides/bat-workflows.md index 9322c8d..6a31199 100644 --- a/docs/guides/bat-workflows.md +++ b/docs/guides/bat-workflows.md @@ -68,7 +68,7 @@ bat parse clear-cache \ bat parse repack --repack-spec /tmp/bat-repack.json ``` -重打包写入独立的 `target_bundle`,逐个操作后由底层 UnityFS patch 实现重建并校验,不允许 source 和 target 相同。 +重打包写入独立的 `target_bundle`,逐个操作后由底层 UnityFS patch 实现重建并校验,不允许 source 和 target 相同。重建会保留已识别的 block 压缩、alignment、目录和未修改对象内容;未知压缩模式或无法证明保真的结构会失败。 ## 翻译工作台与发布 @@ -340,7 +340,8 @@ release,校验工作台与当前 TextUnit 索引的 source/location 一致后 范围内的 TextAsset、TypeTree string field 和 managed-reference string field patch。校验通过后才原子切换 `localized/current`,并在 release manifest 中记录 源/目标 BLAKE3、字节数、patch kind、TextUnit、provider、review、发布时重新计算的 -Glossary QA/override 和 rollback 信息。ZIP 内 bundle 不会被静默改写。 +Glossary QA/override 和 rollback 信息。若 TextUnit 带有 `archive_entry`,发布会在 staging 内验证 ZIP 条目路径,修改并重解析内层 UnityFS 后重写外层 ZIP;路径不安全、内层结构无效或重打包工具失败时不会发布不完整结果。可通过 +`--unzip `、`--zip ` 或对应的 `BAT_UNZIP`、`BAT_ZIP` 配置工具路径。 使用人工编辑的工作台发布: diff --git a/docs/reference/rpc-backend-api.md b/docs/reference/rpc-backend-api.md index ec83c41..0e8f29f 100644 --- a/docs/reference/rpc-backend-api.md +++ b/docs/reference/rpc-backend-api.md @@ -442,6 +442,12 @@ daemon 重启后仍处于 `queued` 或 `running` 的历史任务会被标记为 `data` 会返回 source / patch 或 replacement / target 的 size 与 BLAKE3。`target_path` 不能与输入文件相同。 +`localized.publish` 从当前官方 TextUnit/工作台生成受支持的 UnityFS patch +operation;当 TextUnit 带有 `archive_entry` 时,发布 manifest 的 operation +会记录该可选字段,Rust 发布器会在独立 staging 中校验、重建内层 UnityFS 并 +重写外层 ZIP。ZIP 路径、内层解析或重打包校验失败时整个发布失败,不会只发布 +部分结果。 + 仍关闭的范围:通用 manifest 驱动的发布级 `patch build` / `patch rollback`、复杂 UnityFS 语义编辑、 `unityfs.inspect`、通用 manifest 驱动 release 切换。调用这些规划方法仍返回 `BAT-ERR-700003`。 diff --git a/docs/reports/CURRENT_GAPS.md b/docs/reports/CURRENT_GAPS.md index fe81842..d373af7 100644 --- a/docs/reports/CURRENT_GAPS.md +++ b/docs/reports/CURRENT_GAPS.md @@ -1,6 +1,6 @@ # 当前实现缺口清单 -- **更新时间**:2026-09-04 +- **更新时间**:2026-09-10 - **文档角色**:只记录尚未完成、仍需验证或仍需设计的工作,不重复维护完整实现状态。 - **当前事实**:以源码、测试、稳定契约和 `CURRENT_STATUS.md` 为准。 - **Go 进度**:`GO_STATUS.md` @@ -10,25 +10,27 @@ ## 1. 当前工程缺口 -### G-005:AssetBundle 复杂解析仍未完成 +### G-005:AssetBundle V1 已完成,完整兼容仍未完成 -状态:**部分完成,继续推进** +状态:**V1 已完成(仅限已验证结构),继续推进真实版本和复杂结构** 当前已具备 UnityFS 容器校验、directory 文件提取、serialized file object/type table/TypeTree 元数据、TextAsset、基础 MonoBehaviour 和 -ScriptableObject 字段读取、TextUnit 提取,以及受支持字段的文件级重建。 +ScriptableObject 字段读取、TextUnit 提取,以及受支持字段的文件级 +parse→modify→rebuild→reparse。重建会保留已识别的 block 压缩、alignment、 +directory 形态和未修改对象/字段,并明确拒绝未知压缩或无法证明保真的输入。 仍需完成: - 用真实资源 fixture 覆盖更多 MonoBehaviour、ScriptableObject、Unity 版本差异、 复杂容器和 managed reference registry/map entry 变体。 - 为未知字段补充结构语义;不能把低保真猜测当作已支持格式。 -- 完成发布级复杂对象重打包,并把 bundle、serialized file、path id、class id、 - field path、offset 和 byte size 的定位信息贯通到稳定发布流程。 +- 扩大真实 Unity 版本、复杂容器、未知字段和 managed-reference/map 变体覆盖; + 当前 V1 不等价于任意 AssetBundle 结构的通用重打包。 -现有证据:`crates/bat-assetbundle` 的单元/重建测试、隔离真实 UnityFS 回归和 -`bat-infrastructure` 的解析缓存测试。新增格式覆盖必须同时补真实 fixture、回归测试 -和文档。 +现有证据:`crates/bat-assetbundle` 的单元/压缩/对齐/变长重建测试、隔离真实 UnityFS +回归和 `bat-infrastructure` 的解析缓存、ZIP 内 bundle 发布测试。新增格式覆盖必须 +同时补真实 fixture、回归测试和文档。 ### G-006:通用 Patch 发布仍未完成 @@ -42,7 +44,8 @@ string field、managed-reference string field 写入及 localized publish/rollba 仍需完成: - 通用 manifest 驱动的跨类型 patch build/apply/publish/rollback。 -- 复杂 AssetBundle 重打包和完整翻译文件集合构建。 +- 任意复杂 AssetBundle 重打包和完整翻译文件集合构建;当前 localized publish 已支持 + 可验证 ZIP 内 bundle 的外层 ZIP 重写,但不扩大 UnityFS 结构支持范围。 - 原版 release 与 localized release 双发布后的查询、分发和清理策略。 所有发布产物必须先进入独立 staging,通过完整性校验后再原子发布;失败不得改变 @@ -143,7 +146,7 @@ Rust `bat` 已提供独立项目级 `glossary.sqlite`:term/alias/recommended/a ## 3. 后续推进顺序 -1. 继续 G-005:真实 AssetBundle 样本、复杂字段解析和发布级重打包。 +1. 继续 G-005:更多真实 AssetBundle 样本、复杂字段解析、版本差异和任意结构重打包。 2. 继续 G-006/G-011D:通用 manifest Patch 和双 release 查询/清理策略。 3. 继续 G-011/G-012/G-013/G-014:资源查询、TM/Glossary 扩展和 Provider 扩展体系。 diff --git a/infrastructure/src/bin/bat/app.rs b/infrastructure/src/bin/bat/app.rs index 97770f7..325b3ad 100644 --- a/infrastructure/src/bin/bat/app.rs +++ b/infrastructure/src/bin/bat/app.rs @@ -5098,6 +5098,7 @@ fn sync_command_rpc_method(options: &CliOptions, command_name: &str) -> Option<& && options.config.curl_command == defaults.curl_command && options.config.curl_proxy == defaults.curl_proxy && options.config.unzip_command == defaults.unzip_command + && options.config.zip_command == defaults.zip_command && !options.config.dry_run && !options.config.plan && options.config.audit_local == defaults.audit_local @@ -5351,6 +5352,7 @@ fn build_doctor_report( command_check("curl", &config.curl_command), proxy_check(&config.curl_proxy), command_check("unzip", &config.unzip_command), + command_check("zip", &config.zip_command), ]; let pid_path = daemon_pid_path(state_dir); @@ -6246,6 +6248,8 @@ fn daemon_child_args(options: &CliOptions) -> Vec { } args.push("--unzip".to_string()); args.push(config.unzip_command.to_string_lossy().to_string()); + args.push("--zip".to_string()); + args.push(config.zip_command.to_string_lossy().to_string()); if config.force { args.push("--force".to_string()); } @@ -6575,6 +6579,9 @@ fn apply_bat_env_overrides( if let Some(v) = value("BAT_UNZIP") { options.config.unzip_command = PathBuf::from(v); } + if let Some(v) = value("BAT_ZIP") { + options.config.zip_command = PathBuf::from(v); + } if let Some(v) = value("BAT_TRANSLATION_PROVIDER") { options.translation_provider = Some(v); } @@ -7077,6 +7084,9 @@ fn parse_args_with_env( "--unzip" => { options.config.unzip_command = PathBuf::from(next_option_value(&mut args, &flag)?); } + "--zip" => { + options.config.zip_command = PathBuf::from(next_option_value(&mut args, &flag)?); + } "--dry-run" => { options.config.dry_run = true; options.sync_option_explicit = true; @@ -7744,7 +7754,7 @@ fn parse_args_with_env( CliCommand::Doctor => { if options.sync_option_explicit { return Err(anyhow::anyhow!( - "doctor 只接受 --output、--state-dir、--curl、--proxy 和 --unzip 等诊断参数" + "doctor 只接受 --output、--state-dir、--curl、--proxy、--unzip 和 --zip 等诊断参数" )); } options.progress = false; @@ -7753,7 +7763,7 @@ fn parse_args_with_env( CliCommand::DoctorCas => { if doctor_cas_has_disallowed_sync_options(&options) { return Err(anyhow::anyhow!( - "doctor cas 只接受 --output、--import-repository、--import-cas-root、--import-resource-db、--state-dir、--curl、--proxy 和 --unzip 等诊断参数" + "doctor cas 只接受 --output、--import-repository、--import-cas-root、--import-resource-db、--state-dir、--curl、--proxy、--unzip 和 --zip 等诊断参数" )); } options.progress = false; @@ -7762,7 +7772,7 @@ fn parse_args_with_env( CliCommand::CleanStable => { if options.sync_option_explicit { return Err(anyhow::anyhow!( - "doctor/clean-stable 只接受 --output、--state-dir、--curl、--proxy 和 --unzip 等诊断参数" + "doctor/clean-stable 只接受 --output、--state-dir、--curl、--proxy、--unzip 和 --zip 等诊断参数" )); } options.progress = false; @@ -8181,6 +8191,7 @@ fn doctor_cas_has_disallowed_sync_options(options: &CliOptions) -> bool { allowed.curl_command = options.config.curl_command.clone(); allowed.curl_proxy = options.config.curl_proxy.clone(); allowed.unzip_command = options.config.unzip_command.clone(); + allowed.zip_command = options.config.zip_command.clone(); allowed.import_repository = options.config.import_repository; allowed.import_cas_root = options.config.import_cas_root.clone(); allowed.import_resource_repository_path = @@ -8412,12 +8423,13 @@ fn ensure_command_not_set(command: CliCommand, next: &str) -> anyhow::Result<()> } } -/// 判断 curl/代理/unzip 是否偏离基线。基线是 `config.toml` 和环境变量应用后的 +/// 判断 curl/代理/unzip/zip 是否偏离基线。基线是 `config.toml` 和环境变量应用后的 /// 配置快照,因此只有命令行显式传入才算"非默认"。 fn tools_are_non_default(config: &OfficialUpdateConfig, baseline: &OfficialUpdateConfig) -> bool { config.curl_command != baseline.curl_command || config.curl_proxy != baseline.curl_proxy || config.unzip_command != baseline.unzip_command + || config.zip_command != baseline.zip_command } fn translation_memory_has_unsupported_query_filters(options: &CliOptions) -> bool { diff --git a/infrastructure/src/bin/bat/app_tests.rs b/infrastructure/src/bin/bat/app_tests.rs index 111bfb9..fef7160 100644 --- a/infrastructure/src/bin/bat/app_tests.rs +++ b/infrastructure/src/bin/bat/app_tests.rs @@ -1537,6 +1537,8 @@ fn parses_explicit_source_and_disable_repair() { "--no-repair", "--unzip", "/usr/bin/unzip", + "--zip", + "/usr/bin/zip", ]) .unwrap(); let config = options.config; @@ -1547,6 +1549,7 @@ fn parses_explicit_source_and_disable_repair() { assert!(!config.audit_local); assert!(!config.repair); assert_eq!(config.unzip_command, PathBuf::from("/usr/bin/unzip")); + assert_eq!(config.zip_command, PathBuf::from("/usr/bin/zip")); assert!(matches!( config.server_info_source, Some(OfficialServerInfoSource::OfficialUrl(ref url)) @@ -2226,6 +2229,8 @@ fn daemon_child_args_preserve_sync_options() { "http://127.0.0.1:7890", "--unzip", "/usr/bin/unzip", + "--zip", + "/usr/bin/zip", "--interval", "30m", "--error-retry", @@ -2242,6 +2247,9 @@ fn daemon_child_args_preserve_sync_options() { assert!(args .windows(2) .any(|pair| pair == ["--output", "/tmp/daemon-output"])); + assert!(args + .windows(2) + .any(|pair| pair == ["--zip", "/usr/bin/zip"])); assert!(args .windows(2) .any(|pair| pair == ["--localized-output", "/tmp/daemon-localized"])); diff --git a/infrastructure/src/bin/bat/config_file.rs b/infrastructure/src/bin/bat/config_file.rs index b990826..59767e6 100644 --- a/infrastructure/src/bin/bat/config_file.rs +++ b/infrastructure/src/bin/bat/config_file.rs @@ -61,6 +61,7 @@ import_resource_repository_path = '' curl_command = 'curl' proxy = 'auto' unzip_command = 'unzip' +zip_command = 'zip' download_concurrency = 8 [translation.worker] @@ -138,6 +139,7 @@ struct NetworkSection { curl_command: Option, proxy: Option, unzip_command: Option, + zip_command: Option, download_concurrency: Option, } @@ -356,6 +358,9 @@ impl BatConfigFile { if let Some(value) = self.network.unzip_command.as_ref() { options.config.unzip_command = value.clone(); } + if let Some(value) = self.network.zip_command.as_ref() { + options.config.zip_command = value.clone(); + } if let Some(value) = self.network.download_concurrency { options.config.download_concurrency = value; } @@ -575,6 +580,13 @@ impl BatConfigFile { line_number, )?); } + (SectionPath::Network, "zip_command") => { + self.network.zip_command = Some(parse_required_path( + value, + "network.zip_command", + line_number, + )?); + } (SectionPath::Network, "download_concurrency") => { self.network.download_concurrency = Some(parse_download_concurrency( &parse_scalar_text(value, "network.download_concurrency", line_number)?, @@ -1151,6 +1163,7 @@ import_resource_repository_path = '/srv/resources.sqlite' curl_command = '/usr/bin/curl' proxy = 'http://127.0.0.1:7890' unzip_command = '/usr/bin/unzip' +zip_command = '/usr/bin/zip' download_concurrency = 16 [translation.worker] diff --git a/infrastructure/src/bin/bat/terminal_output.rs b/infrastructure/src/bin/bat/terminal_output.rs index 98a06ae..1be2f44 100644 --- a/infrastructure/src/bin/bat/terminal_output.rs +++ b/infrastructure/src/bin/bat/terminal_output.rs @@ -422,6 +422,7 @@ Sync: --proxy curl proxy override (default: auto from env) --no-proxy Force direct curl connections --unzip unzip executable (default: unzip) + --zip zip executable (default: zip) --dry-run Do not write sync state --plan Include planned URLs in dry-run --force Force download/refresh diff --git a/infrastructure/src/bin/bat/workflow_commands.rs b/infrastructure/src/bin/bat/workflow_commands.rs index 5bab9d8..4eb8ac6 100644 --- a/infrastructure/src/bin/bat/workflow_commands.rs +++ b/infrastructure/src/bin/bat/workflow_commands.rs @@ -390,6 +390,10 @@ pub(super) fn publish_localized_report( official_release_id, Vec::new(), ) + .with_archive_commands( + options.config.unzip_command.clone(), + options.config.zip_command.clone(), + ) .with_operations(operations) .with_force(options.config.force); if let Some(release_id) = localized_release_id { diff --git a/infrastructure/src/localized_patch.rs b/infrastructure/src/localized_patch.rs index 750a68d..406bc9f 100644 --- a/infrastructure/src/localized_patch.rs +++ b/infrastructure/src/localized_patch.rs @@ -1,19 +1,21 @@ //! Localized release publishing for verified UnityFS text patches. use bat_assetbundle::{ - patch_unityfs_field, patch_unityfs_string_field, patch_unityfs_text_asset, FieldPatch, - StringFieldPatch, TextAssetPatch, + patch_unityfs_field, patch_unityfs_string_field, patch_unityfs_text_asset, FieldPatch, Parser, + StringFieldPatch, TextAssetPatch, UnitySerializedField, UnitySerializedValue, }; use serde::{Deserialize, Serialize}; use std::collections::BTreeMap; use std::fs; use std::path::{Path, PathBuf}; +use std::process::Command; use std::time::{SystemTime, UNIX_EPOCH}; use crate::path_security::{ ensure_path_within_root, ensure_safe_directory_path, ensure_safe_file_target, lexical_absolute, read_file_no_symlink, write_file_atomic, STATE_FILE_MODE, }; +use crate::zip_validation::validate_zip_structure; /// Atomic current pointer under the localized output root. pub const LOCALIZED_CURRENT_LINK: &str = "current"; @@ -39,6 +41,8 @@ pub const LOCALIZED_TRANSLATION_STATUS_MANUAL_PROOFREADING_LABEL: &str = "人工 pub struct LocalizedTextAssetPatch { /// Relative path of the UnityFS bundle under the official release. pub bundle_path: String, + /// Relative path of the UnityFS bundle inside an outer ZIP archive. + pub archive_entry: Option, /// TextAsset replacement inside the bundle. pub text_asset: TextAssetPatch, /// TextUnit/provider metadata recorded in the localized manifest. @@ -50,6 +54,8 @@ pub struct LocalizedTextAssetPatch { pub struct LocalizedStringFieldPatch { /// Relative path of the UnityFS bundle under the official release. pub bundle_path: String, + /// Relative path of the UnityFS bundle inside an outer ZIP archive. + pub archive_entry: Option, /// String field replacement inside the bundle. pub string_field: StringFieldPatch, /// TextUnit/provider metadata recorded in the localized manifest. @@ -61,6 +67,8 @@ pub struct LocalizedStringFieldPatch { pub struct LocalizedFieldPatch { /// Relative path of the UnityFS bundle under the official release. pub bundle_path: String, + /// Relative path of the UnityFS bundle inside an outer ZIP archive. + pub archive_entry: Option, /// Semantic field replacement inside the bundle. pub field: FieldPatch, /// TextUnit/provider metadata recorded in the localized manifest. @@ -127,6 +135,10 @@ pub struct LocalizedPatchConfig { pub patches: Vec, /// General UnityFS text/field operations to apply. pub operations: Vec, + /// Unzip executable used for ZIP-inner bundle publication. + pub unzip_command: PathBuf, + /// Zip executable used for ZIP-inner bundle publication. + pub zip_command: PathBuf, } impl LocalizedPatchConfig { @@ -145,6 +157,8 @@ impl LocalizedPatchConfig { force: false, patches, operations: Vec::new(), + unzip_command: PathBuf::from("unzip"), + zip_command: PathBuf::from("zip"), } } @@ -166,6 +180,17 @@ impl LocalizedPatchConfig { self } + /// Sets the archive tools used for ZIP-inner bundle publication. + pub fn with_archive_commands( + mut self, + unzip_command: impl Into, + zip_command: impl Into, + ) -> Self { + self.unzip_command = unzip_command.into(); + self.zip_command = zip_command.into(); + self + } + fn published_release_id(&self) -> &str { self.localized_release_id .as_deref() @@ -193,6 +218,14 @@ impl LocalizedPatchInput { } } + fn archive_entry(&self) -> Option<&str> { + match self { + Self::TextAsset(operation) => operation.archive_entry.as_deref(), + Self::StringField(operation) => operation.archive_entry.as_deref(), + Self::Field(operation) => operation.archive_entry.as_deref(), + } + } + fn apply(&self, input: &[u8]) -> anyhow::Result> { match self { Self::TextAsset(operation) => { @@ -206,7 +239,7 @@ impl LocalizedPatchInput { } fn manifest_operation(&self) -> anyhow::Result { - match self { + let mut operation = match self { Self::TextAsset(operation) => Ok(LocalizedPatchOperation::from_text_asset_patch( &operation.text_asset, operation.metadata.as_ref(), @@ -219,7 +252,9 @@ impl LocalizedPatchInput { &operation.field, operation.metadata.as_ref(), ), - } + }?; + operation.archive_entry = self.archive_entry().map(str::to_string); + Ok(operation) } } @@ -314,6 +349,9 @@ pub struct LocalizedPatchFile { /// One TextAsset patch operation recorded in the localized patch manifest. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct LocalizedPatchOperation { + /// Relative path of the UnityFS bundle inside an outer ZIP archive. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub archive_entry: Option, /// Patch operation kind, for example `unityfs_text_asset`. #[serde(default = "default_patch_operation_kind")] pub patch_kind: String, @@ -499,6 +537,11 @@ impl LocalizedPatchService { .join(&published_release_id); let current_path = config.localized_output_root.join(LOCALIZED_CURRENT_LINK); validate_config(config).map_err(anyhow::Error::msg)?; + let state_path = config + .localized_output_root + .join(LOCALIZED_VERSION_STATE_FILE); + let previous_state_bytes = + read_file_no_symlink(&state_path, "汉化版本状态").map_err(anyhow::Error::msg)?; let previous_current_target = current_symlink_target(¤t_path)?; if let Some(target) = previous_current_target.as_deref() { validate_previous_current_target(&config.localized_output_root, target)?; @@ -514,6 +557,7 @@ impl LocalizedPatchService { !version_existed_before, ¤t_path, previous_current_target.as_ref(), + (&state_path, previous_state_bytes.as_deref()), ) { return Err(anyhow::anyhow!( "{error}; rollback failed: {rollback_error}" @@ -751,14 +795,35 @@ impl LocalizedPatchService { ensure_safe_file_target(&staging, &target, "汉化 patch 输入") .map_err(anyhow::Error::msg)?; let original = fs::read(&target)?; - let mut patched = original.clone(); - let mut manifest_operations = Vec::with_capacity(operations.len()); - for operation in operations { - patched = operation - .apply(&patched) - .map_err(|error| anyhow::anyhow!("{bundle_path}: {error}"))?; - manifest_operations.push(operation.manifest_operation()?); - } + let (patched, manifest_operations) = if operations + .iter() + .any(|operation| operation.archive_entry().is_some()) + { + if operations + .iter() + .any(|operation| operation.archive_entry().is_none()) + { + return Err(anyhow::anyhow!( + "{bundle_path}: 不能混合直接 bundle patch 与 ZIP 内 bundle patch" + )); + } + rewrite_zip_bundle( + &original, + &operations, + &config.unzip_command, + &config.zip_command, + )? + } else { + let mut patched = original.clone(); + let mut manifest_operations = Vec::with_capacity(operations.len()); + for operation in operations { + patched = operation + .apply(&patched) + .map_err(|error| anyhow::anyhow!("{bundle_path}: {error}"))?; + manifest_operations.push(operation.manifest_operation()?); + } + (patched, manifest_operations) + }; if original == patched { return Err(anyhow::anyhow!("patch produced no change: {bundle_path}")); } @@ -800,7 +865,12 @@ impl LocalizedPatchService { "汉化 patch manifest", ) .map_err(anyhow::Error::msg)?; - verify_patch_manifest_files(&config.official_release_root, &staging, &manifest)?; + verify_patch_manifest_files( + &config.official_release_root, + &staging, + &manifest, + &config.unzip_command, + )?; fs::create_dir_all(config.localized_output_root.join(LOCALIZED_VERSIONS_DIR))?; fs::rename(&staging, &version_path)?; switch_current_symlink( @@ -828,6 +898,7 @@ impl LocalizedPatchService { &config.official_release_root, &version_path, ¤t_path, + &config.unzip_command, )?; Ok(LocalizedPatchReport { @@ -954,6 +1025,7 @@ impl LocalizedPatchOperation { ) -> Self { Self::with_metadata( Self { + archive_entry: None, patch_kind: "unityfs_text_asset".to_string(), serialized_file_path: patch.serialized_file_path.clone(), path_id: patch.path_id, @@ -981,6 +1053,7 @@ impl LocalizedPatchOperation { ) -> Self { Self::with_metadata( Self { + archive_entry: None, patch_kind: "unityfs_string_field".to_string(), serialized_file_path: patch.serialized_file_path.clone(), path_id: patch.path_id, @@ -1011,6 +1084,7 @@ impl LocalizedPatchOperation { let replacement = serde_json::to_vec(&patch.replacement)?; Ok(Self::with_metadata( Self { + archive_entry: None, patch_kind: "unityfs_field".to_string(), serialized_file_path: patch.serialized_file_path.clone(), path_id: patch.path_id, @@ -1058,6 +1132,7 @@ fn default_patch_operation_kind() -> String { impl Default for LocalizedPatchOperation { fn default() -> Self { Self { + archive_entry: None, patch_kind: default_patch_operation_kind(), serialized_file_path: String::new(), path_id: 0, @@ -1082,6 +1157,7 @@ fn verify_published_localized_release( official_release_root: &Path, version_path: &Path, current_path: &Path, + unzip_command: &Path, ) -> anyhow::Result { let manifest = read_localized_patch_manifest_at(version_path)?.ok_or_else(|| { anyhow::anyhow!( @@ -1089,8 +1165,12 @@ fn verify_published_localized_release( version_path.join(LOCALIZED_PATCH_MANIFEST_FILE).display() ) })?; - let mut integrity = - verify_patch_manifest_files(official_release_root, version_path, &manifest)?; + let mut integrity = verify_patch_manifest_files( + official_release_root, + version_path, + &manifest, + unzip_command, + )?; integrity.current_points_to_release = current_points_to_version(current_path, version_path)?; if !integrity.current_points_to_release { return Err(anyhow::anyhow!( @@ -1106,6 +1186,7 @@ fn verify_patch_manifest_files( official_release_root: &Path, localized_release_root: &Path, manifest: &LocalizedPatchManifest, + unzip_command: &Path, ) -> anyhow::Result { let mut operation_count = 0usize; for file in &manifest.files { @@ -1151,6 +1232,15 @@ fn verify_patch_manifest_files( file.path )); } + let archive_operations = file + .text_asset_operations + .iter() + .filter(|operation| operation.archive_entry.is_some()) + .collect::>(); + if !archive_operations.is_empty() { + verify_zip_operations(unzip_command, &official_path, &archive_operations, false)?; + verify_zip_operations(unzip_command, &localized_path, &archive_operations, true)?; + } operation_count += file.text_asset_operations.len(); } if manifest.file_count != manifest.files.len() { @@ -1174,6 +1264,316 @@ fn verify_patch_manifest_files( }) } +fn rewrite_zip_bundle( + original: &[u8], + operations: &[LocalizedPatchInput], + unzip_command: &Path, + zip_command: &Path, +) -> anyhow::Result<(Vec, Vec)> { + let archive_temp = tempfile::tempdir()?; + let archive_path = archive_temp.path().join("source.zip"); + fs::write(&archive_path, original)?; + validate_zip_structure(&archive_path).map_err(anyhow::Error::msg)?; + let entries = list_archive_entries(unzip_command, &archive_path)?; + let extract_path = archive_temp.path().join("extract"); + fs::create_dir(&extract_path)?; + + let extract_output = Command::new(unzip_command) + .arg("-q") + .arg("-o") + .arg(&archive_path) + .arg("-d") + .arg(&extract_path) + .output() + .map_err(|error| anyhow::anyhow!("启动 unzip 解包 ZIP 失败:{error}"))?; + ensure_command_success(&extract_output, "解包 ZIP")?; + reject_extracted_symlinks(&extract_path)?; + + let mut manifest_operations = Vec::with_capacity(operations.len()); + let mut patched_entries = BTreeMap::>::new(); + for operation in operations { + let archive_entry = operation.archive_entry().ok_or_else(|| { + anyhow::anyhow!("ZIP patch 缺少 archive_entry:{}", operation.bundle_path()) + })?; + validate_archive_member_name(archive_entry)?; + if !entries.contains(archive_entry) { + return Err(anyhow::anyhow!( + "ZIP 中不存在 archive entry:{}!{}", + operation.bundle_path(), + archive_entry + )); + } + let entry_path = extract_path.join(archive_entry); + ensure_path_within_root(&extract_path, &entry_path).map_err(anyhow::Error::msg)?; + ensure_safe_file_target(&extract_path, &entry_path, "ZIP 内 bundle") + .map_err(anyhow::Error::msg)?; + let current = if let Some(current) = patched_entries.get(archive_entry) { + current.clone() + } else { + fs::read(&entry_path)? + }; + let patched = operation.apply(¤t).map_err(|error| { + anyhow::anyhow!("{}!{}: {error}", operation.bundle_path(), archive_entry) + })?; + fs::write(&entry_path, &patched)?; + patched_entries.insert(archive_entry.to_string(), patched); + manifest_operations.push(operation.manifest_operation()?); + } + + let output_path = archive_temp.path().join("localized.zip"); + let zip_output = Command::new(zip_command) + .current_dir(&extract_path) + .arg("-q") + .arg("-r") + .arg(&output_path) + .arg(".") + .output() + .map_err(|error| anyhow::anyhow!("启动 zip 重打包 ZIP 失败:{error}"))?; + ensure_command_success(&zip_output, "重打包 ZIP")?; + validate_zip_structure(&output_path).map_err(anyhow::Error::msg)?; + let patched = fs::read(output_path)?; + Ok((patched, manifest_operations)) +} + +fn list_archive_entries( + unzip_command: &Path, + archive_path: &Path, +) -> anyhow::Result> { + let output = Command::new(unzip_command) + .arg("-Z1") + .arg(archive_path) + .output() + .map_err(|error| anyhow::anyhow!("启动 unzip 列出 ZIP 条目失败:{error}"))?; + ensure_command_success(&output, "列出 ZIP 条目")?; + let entries = std::str::from_utf8(&output.stdout) + .map_err(|_| anyhow::anyhow!("ZIP 条目列表不是 UTF-8:{}", archive_path.display()))? + .lines() + .map(|line| line.trim_end_matches('\r').to_string()) + .filter(|entry| !entry.is_empty() && !entry.ends_with('/')) + .collect::>(); + for entry in &entries { + validate_archive_member_name(entry)?; + } + Ok(entries) +} + +fn verify_zip_operations( + unzip_command: &Path, + archive_path: &Path, + operations: &[&LocalizedPatchOperation], + verify_replacements: bool, +) -> anyhow::Result<()> { + validate_zip_structure(archive_path).map_err(anyhow::Error::msg)?; + let entries = list_archive_entries(unzip_command, archive_path)?; + for operation in operations { + let archive_entry = operation.archive_entry.as_deref().ok_or_else(|| { + anyhow::anyhow!( + "manifest ZIP operation 缺少 archive_entry:{}", + archive_path.display() + ) + })?; + validate_archive_member_name(archive_entry)?; + if !entries.contains(archive_entry) { + return Err(anyhow::anyhow!( + "manifest ZIP operation 指向不存在的 archive entry:{}!{}", + archive_path.display(), + archive_entry + )); + } + let output = Command::new(unzip_command) + .arg("-p") + .arg(archive_path) + .arg(archive_entry) + .output() + .map_err(|error| anyhow::anyhow!("启动 unzip 读取 ZIP 条目失败:{error}"))?; + ensure_command_success(&output, "读取 ZIP 条目")?; + let parsed = bat_assetbundle::UnityFsParser::new() + .parse(&output.stdout) + .map_err(|error| { + anyhow::anyhow!( + "ZIP 内 UnityFS 重解析失败 {}!{}:{error}", + archive_path.display(), + archive_entry + ) + })?; + if verify_replacements { + verify_archive_operation_replacement(&parsed, operation, archive_entry)?; + } + } + Ok(()) +} + +fn verify_archive_operation_replacement( + parsed: &bat_assetbundle::ParsedAssetBundle, + operation: &LocalizedPatchOperation, + archive_entry: &str, +) -> anyhow::Result<()> { + let Some(serialized_file) = parsed + .serialized_files + .iter() + .find(|file| file.source_path.as_deref() == Some(operation.serialized_file_path.as_str())) + else { + return Err(anyhow::anyhow!( + "ZIP 内 UnityFS manifest serialized file 不存在:{}!{}:{}", + archive_entry, + operation.serialized_file_path, + operation.path_id + )); + }; + + let bytes: Vec = match operation.patch_kind.as_str() { + "unityfs_text_asset" => parsed + .text_assets + .iter() + .find(|asset| { + asset.source_path.as_deref() == Some(operation.serialized_file_path.as_str()) + && asset.path_id == operation.path_id + }) + .map(|asset| asset.bytes.clone()) + .ok_or_else(|| { + anyhow::anyhow!( + "ZIP 内 UnityFS manifest TextAsset 不存在:{}!{}:{}", + archive_entry, + operation.serialized_file_path, + operation.path_id + ) + })?, + "unityfs_string_field" => { + let field_path = operation.field_path.as_deref().ok_or_else(|| { + anyhow::anyhow!( + "ZIP 内 UnityFS string operation 缺少 field_path:{}!{}:{}", + archive_entry, + operation.serialized_file_path, + operation.path_id + ) + })?; + let fields = serialized_file.fields_for_object(operation.path_id)?; + let value = find_archive_field_value(&fields, field_path).ok_or_else(|| { + anyhow::anyhow!( + "ZIP 内 UnityFS manifest 字段不存在:{}!{}:{}:{}", + archive_entry, + operation.serialized_file_path, + operation.path_id, + field_path + ) + })?; + match value { + UnitySerializedValue::String(value) => value.as_bytes().to_vec(), + other => { + return Err(anyhow::anyhow!( + "ZIP 内 UnityFS manifest 字段不是 string:{}!{}:{}:{} ({other:?})", + archive_entry, + operation.serialized_file_path, + operation.path_id, + field_path + )); + } + } + } + // Semantic field replacements retain structure and are already + // verified by the field patch operation before the archive rewrite. + _ => return Ok(()), + }; + + let actual_hash = blake3::hash(&bytes).to_hex().to_string(); + if actual_hash != operation.replacement_blake3 + || bytes.len() as u64 != operation.replacement_bytes + { + return Err(anyhow::anyhow!( + "ZIP 内 UnityFS replacement 校验失败:{}!{}:{},期望 hash={} bytes={},实际 hash={} bytes={}", + archive_entry, + operation.serialized_file_path, + operation.path_id, + operation.replacement_blake3, + operation.replacement_bytes, + actual_hash, + bytes.len() + )); + } + Ok(()) +} + +fn find_archive_field_value<'a>( + fields: &'a [UnitySerializedField], + field_path: &str, +) -> Option<&'a UnitySerializedValue> { + for field in fields { + if field.path == field_path { + return Some(&field.value); + } + let children = match &field.value { + UnitySerializedValue::Object(children) + | UnitySerializedValue::Array(children) + | UnitySerializedValue::Map(children) + | UnitySerializedValue::ManagedReference { + fields: children, .. + } + | UnitySerializedValue::ManagedReferenceRegistry { + fields: children, .. + } => children, + _ => continue, + }; + if let Some(value) = find_archive_field_value(children, field_path) { + return Some(value); + } + } + None +} + +fn ensure_command_success(output: &std::process::Output, action: &str) -> anyhow::Result<()> { + if output.status.success() { + return Ok(()); + } + let stderr = String::from_utf8_lossy(&output.stderr).trim().to_string(); + let stdout = String::from_utf8_lossy(&output.stdout).trim().to_string(); + Err(anyhow::anyhow!( + "{action}失败:{}{}", + stderr, + if stdout.is_empty() { + String::new() + } else { + format!(";输出={stdout}") + } + )) +} + +fn validate_archive_member_name(name: &str) -> anyhow::Result<()> { + let is_windows_absolute = name.as_bytes().get(1) == Some(&b':'); + if name.is_empty() || name.contains('\0') || name.contains('\\') || is_windows_absolute { + return Err(anyhow::anyhow!("ZIP 条目路径无效:{name}")); + } + let path = Path::new(name); + for component in path.components() { + match component { + std::path::Component::Normal(_) | std::path::Component::CurDir => {} + _ => return Err(anyhow::anyhow!("ZIP 条目路径不安全:{name}")), + } + } + Ok(()) +} + +fn reject_extracted_symlinks(root: &Path) -> anyhow::Result<()> { + for entry in fs::read_dir(root)? { + let path = entry?.path(); + let metadata = fs::symlink_metadata(&path)?; + if metadata.file_type().is_symlink() { + return Err(anyhow::anyhow!( + "ZIP 解包结果包含 symlink,拒绝重打包:{}", + path.display() + )); + } + if metadata.is_dir() { + reject_extracted_symlinks(&path)?; + } else if !metadata.is_file() { + return Err(anyhow::anyhow!( + "ZIP 解包结果包含非普通文件:{}", + path.display() + )); + } + } + Ok(()) +} + fn current_symlink_target(current_path: &Path) -> anyhow::Result> { match fs::symlink_metadata(current_path) { Ok(metadata) if metadata.file_type().is_symlink() => Ok(Some(fs::read_link(current_path)?)), @@ -1232,7 +1632,9 @@ fn rollback_failed_publish( remove_version_path: bool, current_path: &Path, previous_current_target: Option<&PathBuf>, + state_restore: (&Path, Option<&[u8]>), ) -> anyhow::Result<()> { + let (state_path, previous_state_bytes) = state_restore; remove_owned_path(staging)?; if remove_version_path { remove_owned_path(version_path)?; @@ -1249,6 +1651,16 @@ fn rollback_failed_publish( }) { restore_current_symlink(localized_output_root, current_path, previous_current_target)?; } + remove_owned_path(state_path)?; + if let Some(previous_state_bytes) = previous_state_bytes { + write_file_atomic( + state_path, + previous_state_bytes, + STATE_FILE_MODE, + "恢复汉化版本状态", + ) + .map_err(anyhow::Error::msg)?; + } Ok(()) } @@ -1415,6 +1827,110 @@ mod tests { use super::*; use tempfile::TempDir; + fn push_u32_be(data: &mut Vec, value: u32) { + data.extend_from_slice(&value.to_be_bytes()); + } + + fn push_u64_be(data: &mut Vec, value: u64) { + data.extend_from_slice(&value.to_be_bytes()); + } + + fn push_u64_le(data: &mut Vec, value: u64) { + data.extend_from_slice(&value.to_le_bytes()); + } + + fn push_u32_le(data: &mut Vec, value: u32) { + data.extend_from_slice(&value.to_le_bytes()); + } + + fn push_i16_le(data: &mut Vec, value: i16) { + data.extend_from_slice(&value.to_le_bytes()); + } + + fn align_vec(data: &mut Vec, alignment: usize) { + let remainder = data.len() % alignment; + if remainder != 0 { + data.resize(data.len() + alignment - remainder, 0); + } + } + + fn push_c_string(data: &mut Vec, value: &str) { + data.extend_from_slice(value.as_bytes()); + data.push(0); + } + + fn synthetic_serialized_text_asset(bytes: &[u8]) -> Vec { + let mut object_data = Vec::new(); + push_u32_le(&mut object_data, 8); + object_data.extend_from_slice(b"Scenario"); + align_vec(&mut object_data, 4); + push_u32_le(&mut object_data, bytes.len() as u32); + object_data.extend_from_slice(bytes); + + let mut metadata = Vec::new(); + metadata.extend_from_slice(b"2021.3.56f2\0"); + metadata.extend_from_slice(&19i32.to_le_bytes()); + metadata.push(0); + metadata.extend_from_slice(&1i32.to_le_bytes()); + metadata.extend_from_slice(&49i32.to_le_bytes()); + metadata.push(0); + push_i16_le(&mut metadata, 0); + metadata.extend_from_slice(&[0; 16]); + metadata.extend_from_slice(&1i32.to_le_bytes()); + align_vec(&mut metadata, 4); + push_u64_le(&mut metadata, 1); + push_u64_le(&mut metadata, 0); + push_u32_le(&mut metadata, object_data.len() as u32); + push_u32_le(&mut metadata, 0); + + let header_len = 48usize; + let data_offset = header_len + metadata.len(); + let file_size = data_offset + object_data.len(); + let mut file = Vec::new(); + push_u32_be(&mut file, metadata.len() as u32); + push_u32_be(&mut file, file_size as u32); + push_u32_be(&mut file, 22); + push_u32_be(&mut file, 0); + file.extend_from_slice(&[0, 0, 0, 0]); + push_u32_be(&mut file, metadata.len() as u32); + push_u64_be(&mut file, file_size as u64); + push_u64_be(&mut file, data_offset as u64); + push_u64_be(&mut file, 0); + file.extend_from_slice(&metadata); + file.extend_from_slice(&object_data); + file + } + + fn synthetic_unityfs(payload: &[u8]) -> Vec { + let mut blocks_info = vec![0; 16]; + blocks_info.extend_from_slice(&1i32.to_be_bytes()); + push_u32_be(&mut blocks_info, payload.len() as u32); + push_u32_be(&mut blocks_info, payload.len() as u32); + blocks_info.extend_from_slice(&0u16.to_be_bytes()); + blocks_info.extend_from_slice(&1i32.to_be_bytes()); + push_u64_be(&mut blocks_info, 0); + push_u64_be(&mut blocks_info, payload.len() as u64); + push_u32_be(&mut blocks_info, 0); + push_c_string(&mut blocks_info, "CAB-asset"); + + let mut file = Vec::new(); + push_c_string(&mut file, "UnityFS"); + push_u32_be(&mut file, 8); + push_c_string(&mut file, "5.x.x"); + push_c_string(&mut file, "2021.3.56f2"); + let total_size_offset = file.len(); + push_u64_be(&mut file, 0); + push_u32_be(&mut file, blocks_info.len() as u32); + push_u32_be(&mut file, blocks_info.len() as u32); + push_u32_be(&mut file, 0); + align_vec(&mut file, 16); + file.extend_from_slice(&blocks_info); + file.extend_from_slice(payload); + let total_size = file.len() as u64; + file[total_size_offset..total_size_offset + 8].copy_from_slice(&total_size.to_be_bytes()); + file + } + #[test] fn localized_manifest_converts_to_generic_patch_manifest() { let source = b"source"; @@ -1434,6 +1950,7 @@ mod tests { localized_bytes: target.len() as u64, byte_delta: target.len() as i64 - source.len() as i64, text_asset_operations: vec![LocalizedPatchOperation { + archive_entry: None, patch_kind: "unityfs_text_asset".to_string(), serialized_file_path: "CAB-asset".to_string(), path_id: 1, @@ -1489,6 +2006,7 @@ mod tests { }; let input = LocalizedPatchInput::TextAsset(LocalizedTextAssetPatch { bundle_path: "Bundles/file.bundle".to_string(), + archive_entry: None, text_asset: TextAssetPatch::new("CAB-asset", 1, b"target".to_vec()), metadata: Some(LocalizedPatchOperationMetadata { text_unit_id: "unit-1".to_string(), @@ -1512,6 +2030,69 @@ mod tests { ); } + #[cfg(unix)] + #[test] + fn publishes_and_reparses_bundle_nested_in_zip() { + let temp = TempDir::new().unwrap(); + let official = temp.path().join("official-release"); + let localized = temp.path().join("localized"); + let zip_source = temp.path().join("zip-source"); + fs::create_dir_all(zip_source.join("nested")).unwrap(); + fs::create_dir_all(official.join("Bundles")).unwrap(); + fs::write( + zip_source.join("nested/CAB-asset"), + synthetic_unityfs(&synthetic_serialized_text_asset(b"old")), + ) + .unwrap(); + let archive_path = official.join("Bundles/catalog.zip"); + let zip_status = std::process::Command::new("zip") + .current_dir(&zip_source) + .arg("-q") + .arg("-r") + .arg(&archive_path) + .arg(".") + .status() + .unwrap(); + assert!(zip_status.success()); + + let operation = LocalizedPatchInput::TextAsset(LocalizedTextAssetPatch { + bundle_path: "Bundles/catalog.zip".to_string(), + archive_entry: Some("nested/CAB-asset".to_string()), + text_asset: TextAssetPatch::new("CAB-asset", 1, b"translated".to_vec()), + metadata: None, + }); + let report = LocalizedPatchService::new() + .publish( + &LocalizedPatchConfig::new(&official, &localized, "release-1", Vec::new()) + .with_archive_commands("unzip", "zip") + .with_operations(vec![operation]), + ) + .unwrap(); + + let localized_archive = report.version_path.join("Bundles/catalog.zip"); + let output = std::process::Command::new("unzip") + .arg("-p") + .arg(&localized_archive) + .arg("nested/CAB-asset") + .output() + .unwrap(); + assert!(output.status.success()); + let parsed = bat_assetbundle::UnityFsParser::new() + .parse(&output.stdout) + .unwrap(); + assert_eq!(parsed.text_assets[0].bytes, b"translated"); + assert_eq!( + report.manifest.files[0].text_asset_operations[0] + .archive_entry + .as_deref(), + Some("nested/CAB-asset") + ); + assert_eq!( + fs::read_link(report.current_path).unwrap(), + PathBuf::from("versions/release-1") + ); + } + #[cfg(unix)] #[test] fn publishes_a_separate_localized_release_atomically() { @@ -1709,6 +2290,7 @@ mod tests { "release-1", vec![LocalizedTextAssetPatch { bundle_path: "Bundles/bad.bundle".to_string(), + archive_entry: None, text_asset: TextAssetPatch::new("CAB-bad", 1, b"replacement".to_vec()), metadata: None, }], @@ -1726,4 +2308,49 @@ mod tests { .exists()); assert!(!localized.join(LOCALIZED_CURRENT_LINK).exists()); } + + #[cfg(unix)] + #[test] + fn failed_publish_rollback_restores_previous_state_file() { + use std::os::unix::fs::symlink; + + let temp = TempDir::new().unwrap(); + let localized = temp.path().join("localized"); + let versions = localized.join(LOCALIZED_VERSIONS_DIR); + let staging = localized.join(LOCALIZED_STAGING_DIR).join("release-new"); + let previous_version = versions.join("release-old"); + let failed_version = versions.join("release-new"); + let current = localized.join(LOCALIZED_CURRENT_LINK); + let state_path = localized.join(LOCALIZED_VERSION_STATE_FILE); + let previous_state = br#"{"state_version":1,"status":"localized"}"#; + + fs::create_dir_all(&previous_version).unwrap(); + fs::create_dir_all(&failed_version).unwrap(); + fs::create_dir_all(&staging).unwrap(); + fs::write(&state_path, previous_state).unwrap(); + symlink( + Path::new(LOCALIZED_VERSIONS_DIR).join("release-new"), + ¤t, + ) + .unwrap(); + + rollback_failed_publish( + &localized, + &staging, + &failed_version, + true, + ¤t, + Some(&PathBuf::from("versions/release-old")), + (&state_path, Some(previous_state)), + ) + .unwrap(); + + assert!(!failed_version.exists()); + assert!(!staging.exists()); + assert_eq!( + fs::read_link(¤t).unwrap(), + PathBuf::from("versions/release-old") + ); + assert_eq!(fs::read(&state_path).unwrap(), previous_state); + } } diff --git a/infrastructure/src/official_update.rs b/infrastructure/src/official_update.rs index 96e90bc..1e2cbe2 100644 --- a/infrastructure/src/official_update.rs +++ b/infrastructure/src/official_update.rs @@ -126,6 +126,8 @@ pub struct OfficialUpdateConfig { pub download_concurrency: usize, /// Unzip command used when a metadata change requires GameMainConfig parsing. pub unzip_command: PathBuf, + /// Zip command used when publishing localized bundles nested in ZIP archives. + pub zip_command: PathBuf, /// Dry run reports decisions and optional plan URLs without writing sync state. pub dry_run: bool, /// Include full download URLs when dry-running. @@ -162,6 +164,7 @@ impl Default for OfficialUpdateConfig { curl_proxy: CurlProxyConfig::default(), download_concurrency: DEFAULT_DOWNLOAD_CONCURRENCY, unzip_command: PathBuf::from("unzip"), + zip_command: PathBuf::from("zip"), dry_run: false, plan: false, force: false, diff --git a/infrastructure/src/translation_workflow.rs b/infrastructure/src/translation_workflow.rs index 676521c..a226730 100644 --- a/infrastructure/src/translation_workflow.rs +++ b/infrastructure/src/translation_workflow.rs @@ -475,11 +475,10 @@ pub fn validate_translation_workbench_with_glossary_path( } changed_entries += 1; let source_kind = normalized_text_source_kind(entry.text_source_kind.as_deref()); - let is_publishable = entry.archive_entry.is_none() - && matches!( - source_kind.as_deref(), - Some("textasset" | "typetreefield" | "managedreferencefield") - ); + let is_publishable = matches!( + source_kind.as_deref(), + Some("textasset" | "typetreefield" | "managedreferencefield") + ); if is_publishable { let serialized_file = entry .serialized_file @@ -579,10 +578,7 @@ fn validate_current_glossary_qa( } /// Converts reviewed entries to localized patch operations supported by the -/// current UnityFS write layer. -/// -/// ZIP-inner bundles are intentionally rejected here because they require a -/// separate archive rewrite boundary. +/// current UnityFS write layer and its ZIP rewrite boundary. pub fn localized_patch_operations( resource_root: &Path, workbench: &TranslationWorkbench, @@ -654,16 +650,11 @@ pub fn localized_patch_operations_with_glossary_path( entry.id )); }; - if entry.archive_entry.is_some() { - return Err(anyhow::anyhow!( - "TextUnit {} 位于 zip archive entry,当前 publish-localized 不支持直接修改 zip 内 bundle", - entry.id - )); - } let source_kind = normalized_text_source_kind(entry.text_source_kind.as_deref()); let field_path = entry.field_path.clone(); if !seen.insert(( entry.destination.clone(), + entry.archive_entry.clone(), serialized_file.clone(), path_id, field_path.clone(), @@ -688,6 +679,7 @@ pub fn localized_patch_operations_with_glossary_path( patch.expected_name = entry.asset_name.clone(); operations.push(LocalizedPatchInput::TextAsset(LocalizedTextAssetPatch { bundle_path: entry.destination.clone(), + archive_entry: entry.archive_entry.clone(), text_asset: patch, metadata, })); @@ -702,6 +694,7 @@ pub fn localized_patch_operations_with_glossary_path( operations.push(LocalizedPatchInput::StringField( LocalizedStringFieldPatch { bundle_path: entry.destination.clone(), + archive_entry: entry.archive_entry.clone(), string_field: StringFieldPatch { serialized_file_path: serialized_file, path_id, @@ -771,12 +764,6 @@ pub fn localized_text_asset_patches( entry.id )); }; - if entry.archive_entry.is_some() { - return Err(anyhow::anyhow!( - "TextUnit {} 位于 zip archive entry,当前 publish-localized 不支持直接修改 zip 内 bundle", - entry.id - )); - } if normalized_text_source_kind(entry.text_source_kind.as_deref()).as_deref() != Some("textasset") { @@ -785,7 +772,12 @@ pub fn localized_text_asset_patches( entry.id )); } - if !seen.insert((entry.destination.clone(), serialized_file.clone(), path_id)) { + if !seen.insert(( + entry.destination.clone(), + entry.archive_entry.clone(), + serialized_file.clone(), + path_id, + )) { return Err(anyhow::anyhow!( "翻译工作台包含重复 patch 目标:{}", entry.id @@ -811,6 +803,7 @@ pub fn localized_text_asset_patches( } patches.push(LocalizedTextAssetPatch { bundle_path: entry.destination.clone(), + archive_entry: entry.archive_entry.clone(), text_asset: patch, metadata: Some(localized_patch_metadata( entry,