From d3e4525a89452a095aaded6a6ca5c9b3e5f22fbf Mon Sep 17 00:00:00 2001 From: Tim Thomas Date: Thu, 20 Aug 2026 11:04:36 -0500 Subject: [PATCH 1/2] Add hybrid-duplication-audit skill Audits asset duplication across the AssetBundle / ContentDirectory boundary in a hybrid Addressables 4.x build. Runs one `analyze` combining the Addressables build layout report with the content directory build output, then matches source-asset paths across both - catching cases a CRC-based diff misses, such as a shader built with a different variant set on each side (which is also why Addressables' own DuplicatedAssetCount can read 0 even when real duplication exists). --- AGENTS.md | 4 + Documentation/analyzer.md | 7 + README.md | 5 + Skills/hybrid-duplication-audit/SKILL.md | 145 +++++++++++ .../scripts/Compare-HybridDuplication.ps1 | 229 ++++++++++++++++++ 5 files changed, 390 insertions(+) create mode 100644 Skills/hybrid-duplication-audit/SKILL.md create mode 100644 Skills/hybrid-duplication-audit/scripts/Compare-HybridDuplication.ps1 diff --git a/AGENTS.md b/AGENTS.md index c03ff9a..e526506 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -138,6 +138,10 @@ CLI entry point is `UnityDataTool/Program.cs` using System.CommandLine. Per-comm * Any database schema change (new or changed tables, views, or columns) must bump `PRAGMA user_version` in `Analyzer/Resources/Init.sql` and extend the version-history comment above it. * Analysis of additional file formats could be added, for example AssetBundle manifest files by following the pattern of Addressables build layout files are handled. +### Skills + +`Skills/` holds portable Claude Code skills (a `SKILL.md` plus any scripts they need) that wrap common analysis tasks. See `Skills/hybrid-duplication-audit/SKILL.md` for an example — it audits asset duplication across the AssetBundle/content-directory boundary in a hybrid Addressables build. + ### Other Extensions The UnityFileSystem API and UnityBinaryFormat parsing can be useful for other analysis. The "dump", "analyze" and "serialized-file" commands can be considered reference examples of how to use those lower level tools. diff --git a/Documentation/analyzer.md b/Documentation/analyzer.md index 099a2c6..edf317c 100644 --- a/Documentation/analyzer.md +++ b/Documentation/analyzer.md @@ -59,6 +59,13 @@ duplicated assets. It also lists all the AssetBundles where the asset was found. If the `--skip-crc` option is used, there will be a lot of false positives in that view. Otherwise, it should be very accurate because CRCs are used to determine if objects are identical. +This view (and Addressables' own `DuplicatedAssetCount`) can under-report for a hybrid build that +mixes AssetBundle groups with a `ContentDirectoryGroupSchema` group: a source asset built into both +forms doesn't necessarily produce byte-identical objects (a shader's variants, for example, are +stripped independently on each side), so CRC matching can miss it entirely. See +[Skills/hybrid-duplication-audit](../Skills/hybrid-duplication-audit/SKILL.md) for a query that +matches by source asset path instead, which catches this case. + ## assetbundle_asset_view (AssetBundleProcessor) Lists the assets that were explicitly assigned to AssetBundles, one row per entry in the AssetBundle diff --git a/README.md b/README.md index aeffbc3..7712b81 100644 --- a/README.md +++ b/README.md @@ -103,6 +103,11 @@ flowchart TD * UnityBinaryFormat: C# parsers and helpers for reading data out of Unity Archives and SerializedFiles. * UnityDataModels: shared C# models for the reading JSON format files produced by the build (Addressables BuildLayout.json, Content Directory ContentLayout.json). +**Skills** +* [Skills/hybrid-duplication-audit](Skills/hybrid-duplication-audit/SKILL.md): a Claude Code skill + that audits asset duplication across the AssetBundle/content-directory boundary in hybrid + Addressables builds. + ## Purpose of UnityFileSystemApi UnityFileSystemApi is compiled from the Unity source code and exposes the core functionality to open and read the Unity Archive and Serialized File formats as a flexible, performant library. It exposes the ability to navigate the TypeTrees inside a SerializedFile so objects can be read generically, without hardcoded type knowledge. diff --git a/Skills/hybrid-duplication-audit/SKILL.md b/Skills/hybrid-duplication-audit/SKILL.md new file mode 100644 index 0000000..033a6b0 --- /dev/null +++ b/Skills/hybrid-duplication-audit/SKILL.md @@ -0,0 +1,145 @@ +--- +name: hybrid-duplication-audit +description: Audits asset duplication across the AssetBundle / ContentDirectory boundary in a hybrid Addressables 4.x build, where some groups build to .bundle files and others build through BuildPipeline.BuildContentDirectory. Use when asked to compare duplication in a hybrid project, find what's duplicated between bundles and the content directory, explain why Addressables' own DuplicatedAssetCount reads 0 or looks too low despite shared shaders, textures, or meshes across bundle and content-directory groups, or size the "hybrid tax" of assets baked twice because an AssetBundle cannot reference content-directory content. Runs one UnityDataTool analyze combining the Addressables build layout report with the content directory build output, then matches source-asset paths across both to find what's genuinely duplicated — including cases a plain CRC/hash diff misses, such as the same shader built with a different variant set on each side. +compatibility: Requires UnityDataTool (github.com/Unity-Technologies/UnityDataTools) built from a version with ContentLayout.json support, sqlite3 on PATH, and a completed hybrid Addressables 4.x build (at least one AssetBundle-producing group and one ContentDirectoryGroupSchema group). +--- + +# Hybrid duplication audit + +## What this audits + +A hybrid Addressables build produces two independent content builds +that only merge at the catalog: one for groups using +`BundledAssetGroupSchema` (AssetBundles), one for groups using +`ContentDirectoryGroupSchema` (a content directory, `BuildPipeline. +BuildContentDirectory`). An AssetBundle cannot reference content that +lives in a content directory, so any source asset both sides need +gets a full copy baked into the bundle. This skill finds and sizes +exactly that copy. + +## Why Addressables' own report misses it + +Addressables' build layout JSON has a top-level `DuplicatedAssetCount`, +but it only scans AssetBundle-to-AssetBundle duplication. A copy that +lives in the content directory is invisible to it, so it reads low or +zero even when real cross-boundary duplication exists. + +UnityDataTool's own `view_potential_duplicates` (see +[analyzer.md](../../Documentation/analyzer.md)) doesn't fill that gap +either, because it matches by object CRC. A source asset built into +both forms doesn't necessarily produce byte-identical objects — a +shader, for example, gets its variants stripped independently on each +side, so the two copies can have different CRCs despite being the +same shader. Matching by **source asset path** instead of CRC catches +this; that's what this skill does. + +## When to use this + +- "Compare duplication in a hybrid project" +- "What's duplicated between the bundles and the content directory?" +- "Why does DuplicatedAssetCount say 0 when I have shared shaders?" +- Sizing the cost of moving a group from local (content directory) to + remote (AssetBundle), or vice versa. + +## Prerequisites + +- A completed Addressables build containing at least one + AssetBundle-producing group and one `ContentDirectoryGroupSchema` + group. +- `UnityDataTool` built (`dotnet build -c Release` in this repo) — + needs a version with `ContentLayout.json` ingestion + (`content_layout_*` tables; check with + `UnityDataTool --version`, or that `Documentation/contentlayout.md` + exists in your checkout). +- `sqlite3` on PATH. + +## Running it + +```powershell +Skills/hybrid-duplication-audit/scripts/Compare-HybridDuplication.ps1 -ProjectRoot "" +``` + +By default it locates everything it needs on its own: the mirrored +`Library/com.unity.addressables/buildlayout.json` (Addressables keeps +this in sync with its latest build), the one content directory output +folder under `Library/com.unity.addressables/aa`, and +`Library/BuildHistory`. Override any of them with `-BuildLayout`, +`-ContentDirectory`, or `-BuildHistory` — useful for auditing an older +build, or when more than one platform has been built (the script +requires `-ContentDirectory` explicitly in that case, since it can't +guess which platform you mean). + +**When overriding, make sure `-BuildLayout` and `-ContentDirectory` +came from the same build.** Addressables can rebuild only some groups +at a time, so the two can legitimately drift out of sync — nothing +ties an Addressables build layout to a specific content-directory +build the way `--build-history` ties a content directory to its own +`ContentLayout.json`. Pairing a stale report with a newer content +directory (or vice versa) won't error; it'll just report a wrong, +usually much larger, "duplicate" total. The defaults are always safe +in this respect, since both point at whatever is currently on disk. + +Other flags: `-ToolPath` (if `UnityDataTool` isn't on PATH or set via +`UNITYDATATOOL_PATH`), `-KeepDatabase` (preserve the generated +database for follow-up queries), `-MaxRows`. + +## Reading the output + +- **Cross-boundary duplicates** — the source assets built into both + forms, with the bytes they cost on the bundle side. Every row here + is pure hybrid-layout tax: bytes that exist only because the two + builds can't share. +- **Summary** — bundle-side and content-directory-side total payload, + the duplicated total, and what percentage of the bundle payload it + represents. +- **`DuplicatedAssetCount`** — printed for contrast, with the caveat + above. Expect it to understate the real total, sometimes to zero. + +## If it reports "this is not a hybrid build" + +The build layout has no AssetBundle groups, or the content directory +has no source assets (or wasn't paired with its `ContentLayout.json` +— check that `-BuildHistory` points at the project's actual build +history folder). Both counts, printed in the error, need to be +non-zero for the audit to mean anything. + +## Installing this skill into a Unity project + +Claude Code doesn't discover skills living in an unrelated repo. Copy +or symlink this whole folder into the project you want to audit: + +``` +/.claude/skills/hybrid-duplication-audit/ +``` + +Once installed there, `-ToolPath` (or `UNITYDATATOOL_PATH`) will +usually be needed explicitly, since the script's own +UnityDataTools-checkout fallback won't resolve outside this repo. + +## Going deeper + +The core query, if you want to adapt it directly instead of running +the script: + +```sql +WITH bundle_assets AS ( + SELECT asset_path, serialized_size + streamed_size AS bytes + FROM addressables_build_explicit_assets + UNION ALL + SELECT asset_path, serialized_size + streamed_size AS bytes + FROM addressables_build_data_from_other_assets +), +content_dir_assets AS ( + SELECT DISTINCT asset_path FROM content_layout_source_assets +) +SELECT b.asset_path, SUM(b.bytes) AS bundle_bytes, COUNT(*) AS instances +FROM bundle_assets b +WHERE b.asset_path IN (SELECT asset_path FROM content_dir_assets) +GROUP BY b.asset_path +ORDER BY bundle_bytes DESC; +``` + +Run with `-KeepDatabase` and query it directly with `sqlite3`, or see +[analyze-examples-contentlayout.md](../../Documentation/analyze-examples-contentlayout.md) +and [addressables-build-reports.md](../../Documentation/addressables-build-reports.md) +for the surrounding schema. diff --git a/Skills/hybrid-duplication-audit/scripts/Compare-HybridDuplication.ps1 b/Skills/hybrid-duplication-audit/scripts/Compare-HybridDuplication.ps1 new file mode 100644 index 0000000..17dd3ed --- /dev/null +++ b/Skills/hybrid-duplication-audit/scripts/Compare-HybridDuplication.ps1 @@ -0,0 +1,229 @@ +# Finds source assets duplicated across the AssetBundle / ContentDirectory boundary of a +# hybrid Addressables 4.x build (some groups build to .bundle files, others build through +# BuildPipeline.BuildContentDirectory / ContentDirectoryGroupSchema). +# +# It requires that UnityDataTool has been built (see ../../../README.md, "How to Build") and +# that sqlite3 is installed and available on PATH, exactly like Scripts/comparebuilds.ps1 and +# Scripts/comparebundles.ps1 in this repo. +# +# See ../SKILL.md for the concept, the query this script runs, and how to read the output. +# +# DISCLAIMER: +# This script is provided "as-is," without any warranty of any kind, express or implied. +# By using this script, you agree that you understand its purpose and that you use it entirely +# at your own risk. Always review and test this script in a safe environment before applying it +# to a production system. + +param ( + [Parameter(HelpMessage = "Root of the Unity project to audit")] + [string]$ProjectRoot = ".", + + [Parameter(HelpMessage = "Path to UnityDataTool.exe. Defaults to UNITYDATATOOL_PATH, then PATH, then the build output of this checkout")] + [string]$ToolPath, + + [Parameter(HelpMessage = "Addressables build layout report to read. Defaults to the mirrored Library/com.unity.addressables/buildlayout.json, which Addressables keeps in sync with its latest build")] + [string]$BuildLayout, + + [Parameter(HelpMessage = "Content directory build output folder (the one containing BuildManifestHash.txt). Auto-detected under Library/com.unity.addressables/aa when there is exactly one")] + [string]$ContentDirectory, + + [Parameter(HelpMessage = "Project's build history folder")] + [string]$BuildHistory, + + [Parameter(HelpMessage = "Output database path. Defaults to a temp file, deleted afterwards unless -KeepDatabase is set")] + [string]$Database, + + [Parameter(HelpMessage = "Keep the generated database instead of deleting it, for follow-up queries")] + [switch]$KeepDatabase, + + [Parameter(HelpMessage = "Maximum number of duplicate rows to print")] + [int]$MaxRows = 50 +) + +function Resolve-ToolPath { + param([string]$Explicit) + + if ($Explicit) { return $Explicit } + if ($env:UNITYDATATOOL_PATH) { return $env:UNITYDATATOOL_PATH } + + $onPath = Get-Command "UnityDataTool" -ErrorAction SilentlyContinue + if (-not $onPath) { $onPath = Get-Command "UnityDataTool.exe" -ErrorAction SilentlyContinue } + if ($onPath) { return $onPath.Source } + + # Fall back to this checkout's own build output (only valid while the skill is still + # inside a UnityDataTools clone; once copied into another project's .claude/skills, pass + # -ToolPath or set UNITYDATATOOL_PATH instead). + $repoRelative = Join-Path $PSScriptRoot "..\..\..\UnityDataTool\bin\Release\net9.0\UnityDataTool.exe" + if (Test-Path $repoRelative) { return (Resolve-Path $repoRelative).Path } + + return $null +} + +$resolvedToolPath = Resolve-ToolPath -Explicit $ToolPath +if (-not $resolvedToolPath -or -not (Test-Path $resolvedToolPath)) { + Write-Error "UnityDataTool executable not found. Tried: -ToolPath, UNITYDATATOOL_PATH, PATH, and this checkout's own build output. Build it with 'dotnet build -c Release' or pass -ToolPath explicitly." + exit 1 +} + +if (-not (Test-Path $ProjectRoot)) { + Write-Error "Project root '$ProjectRoot' not found." + exit 1 +} +$ProjectRoot = (Resolve-Path $ProjectRoot).Path + +if (-not $BuildLayout) { + $BuildLayout = Join-Path $ProjectRoot "Library\com.unity.addressables\buildlayout.json" +} +if (-not (Test-Path $BuildLayout)) { + Write-Error "Addressables build layout report not found at '$BuildLayout'. Build the project with Addressables first, or pass -BuildLayout explicitly." + exit 1 +} + +if (-not $ContentDirectory) { + $candidates = @(Get-ChildItem -Path (Join-Path $ProjectRoot "Library\com.unity.addressables\aa") ` + -Recurse -Filter "BuildManifestHash.txt" -ErrorAction SilentlyContinue | + ForEach-Object { $_.Directory.FullName }) + + if ($candidates.Count -eq 0) { + Write-Error "No content directory build output found under Library/com.unity.addressables/aa. Pass -ContentDirectory explicitly, or this project may not have a ContentDirectoryGroupSchema group." + exit 1 + } + if ($candidates.Count -gt 1) { + Write-Error "More than one content directory build output found (multiple platforms built?): $($candidates -join ', '). Pass -ContentDirectory to pick one." + exit 1 + } + $ContentDirectory = $candidates[0] +} +if (-not (Test-Path $ContentDirectory)) { + Write-Error "Content directory build output '$ContentDirectory' not found." + exit 1 +} + +if (-not $BuildHistory) { + $BuildHistory = Join-Path $ProjectRoot "Library\BuildHistory" +} +if (-not (Test-Path $BuildHistory)) { + Write-Error "Build history folder '$BuildHistory' not found. It is required to pair the content directory output with its ContentLayout.json." + exit 1 +} + +$deleteDatabaseAfter = $false +if (-not $Database) { + $Database = Join-Path ([System.IO.Path]::GetTempPath()) "hybrid-duplication-$([guid]::NewGuid()).db" + $deleteDatabaseAfter = -not $KeepDatabase +} + +Write-Output "Analyzing:" +Write-Output " Build layout: $BuildLayout" +Write-Output " Content directory: $ContentDirectory" +Write-Output " Build history: $BuildHistory" +Write-Output "" + +& $resolvedToolPath analyze $BuildLayout $ContentDirectory --build-history $BuildHistory -o $Database +if ($LASTEXITCODE -ne 0) { + Write-Error "UnityDataTool analyze failed (exit code $LASTEXITCODE)." + exit $LASTEXITCODE +} + +# Hybrid check: both a bundle-producing group and a content-directory build must be present, +# otherwise the intersection below is meaningless. +$bundleCount = [int](sqlite3 $Database "SELECT COUNT(*) FROM addressables_build_bundles;") +$hasLayoutTable = [int](sqlite3 $Database "SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='content_layout_source_assets';") +$sourceAssetCount = 0 +if ($hasLayoutTable -gt 0) { + $sourceAssetCount = [int](sqlite3 $Database "SELECT COUNT(*) FROM content_layout_source_assets;") +} + +if ($bundleCount -eq 0 -or $sourceAssetCount -eq 0) { + Write-Output "This is not a hybrid build:" + Write-Output " AssetBundle groups in the build layout: $bundleCount" + Write-Output " Source assets in the content directory: $sourceAssetCount" + Write-Output "" + Write-Output "Both must be non-zero to audit cross-boundary duplication." + if ($deleteDatabaseAfter) { Remove-Item $Database -ErrorAction SilentlyContinue } + exit 2 +} + +# The one query that answers "which source assets were built into both forms": bundle-side +# asset paths (explicit + implicit) intersected with the content directory's source assets. +# Matching by asset_path, not by object CRC, is what makes this catch cases a plain CRC diff +# (e.g. view_potential_duplicates, see Documentation/analyzer.md) would miss -- the same +# source asset can be built with a different variant set on each side, so the two copies do +# not have the same CRC despite being duplicates of the same asset. +$duplicatesQuery = @" +WITH bundle_assets AS ( + SELECT asset_path, serialized_size + streamed_size AS bytes + FROM addressables_build_explicit_assets + UNION ALL + SELECT asset_path, serialized_size + streamed_size AS bytes + FROM addressables_build_data_from_other_assets +), +content_dir_assets AS ( + SELECT DISTINCT asset_path FROM content_layout_source_assets +) +SELECT b.asset_path, SUM(b.bytes) AS bundle_bytes, COUNT(*) AS instances +FROM bundle_assets b +WHERE b.asset_path IN (SELECT asset_path FROM content_dir_assets) +GROUP BY b.asset_path +ORDER BY bundle_bytes DESC +LIMIT $MaxRows; +"@ + +$totalsQuery = @" +WITH bundle_assets AS ( + SELECT serialized_size + streamed_size AS bytes FROM addressables_build_explicit_assets + UNION ALL + SELECT serialized_size + streamed_size AS bytes FROM addressables_build_data_from_other_assets +) +SELECT SUM(bytes) FROM bundle_assets; +"@ + +$contentDirTotalQuery = "SELECT SUM(size) FROM content_layout_binary_artifacts WHERE category != 'manifest';" + +$duplicateBytesQuery = @" +WITH bundle_assets AS ( + SELECT asset_path, serialized_size + streamed_size AS bytes FROM addressables_build_explicit_assets + UNION ALL + SELECT asset_path, serialized_size + streamed_size AS bytes FROM addressables_build_data_from_other_assets +) +SELECT COALESCE(SUM(bytes), 0) +FROM bundle_assets +WHERE asset_path IN (SELECT DISTINCT asset_path FROM content_layout_source_assets); +"@ + +$bundleTotalBytes = [int64](sqlite3 $Database $totalsQuery) +$contentDirTotalBytes = [int64](sqlite3 $Database $contentDirTotalQuery) +$duplicateBytes = [int64](sqlite3 $Database $duplicateBytesQuery) + +$declaredDuplicateCount = $null +try { + $layoutJson = Get-Content $BuildLayout -Raw | ConvertFrom-Json + $declaredDuplicateCount = $layoutJson.DuplicatedAssetCount +} catch { + Write-Output "(could not read DuplicatedAssetCount from '$BuildLayout': $_)" +} + +Write-Output "=== Cross-boundary duplicates (built into both a bundle and the content directory) ===" +sqlite3 $Database ".mode column" ".headers on" $duplicatesQuery +Write-Output "" + +Write-Output "=== Summary ===" +Write-Output ("Bundle-side asset payload: {0,15:N0} bytes" -f $bundleTotalBytes) +Write-Output ("Content directory payload: {0,15:N0} bytes" -f $contentDirTotalBytes) +Write-Output ("Cross-boundary duplicated: {0,15:N0} bytes" -f $duplicateBytes) +if ($bundleTotalBytes -gt 0) { + $pct = [math]::Round(100.0 * $duplicateBytes / $bundleTotalBytes, 1) + Write-Output " = $pct% of the bundle-side payload" +} +if ($null -ne $declaredDuplicateCount) { + Write-Output "" + Write-Output "Addressables' own DuplicatedAssetCount for this build: $declaredDuplicateCount" + Write-Output "(That count only scans AssetBundle-to-AssetBundle duplication -- it cannot see a copy that lives in the content directory, so it will read low or zero even when the total above is not.)" +} + +if ($deleteDatabaseAfter) { + Remove-Item $Database -ErrorAction SilentlyContinue +} else { + Write-Output "" + Write-Output "Database kept at: $Database" +} From 289079f33dae982d8e48dfdbc8e4575e5ed4aa0f Mon Sep 17 00:00:00 2001 From: Tim Thomas Date: Thu, 20 Aug 2026 11:57:46 -0500 Subject: [PATCH 2/2] Add bash counterpart of the hybrid-duplication-audit script Linux/macOS equivalent of Compare-HybridDuplication.ps1 - same defaults, same queries, same output. Verified against this session's build data: reproduces the PowerShell version's result exactly. --- Skills/hybrid-duplication-audit/SKILL.md | 38 ++- .../scripts/compare-hybrid-duplication.sh | 248 ++++++++++++++++++ 2 files changed, 274 insertions(+), 12 deletions(-) create mode 100755 Skills/hybrid-duplication-audit/scripts/compare-hybrid-duplication.sh diff --git a/Skills/hybrid-duplication-audit/SKILL.md b/Skills/hybrid-duplication-audit/SKILL.md index 033a6b0..010882e 100644 --- a/Skills/hybrid-duplication-audit/SKILL.md +++ b/Skills/hybrid-duplication-audit/SKILL.md @@ -55,21 +55,32 @@ this; that's what this skill does. ## Running it +Two equivalent scripts, same defaults, same queries, same output — +pick the one for your platform: + ```powershell +# Windows Skills/hybrid-duplication-audit/scripts/Compare-HybridDuplication.ps1 -ProjectRoot "" ``` -By default it locates everything it needs on its own: the mirrored -`Library/com.unity.addressables/buildlayout.json` (Addressables keeps -this in sync with its latest build), the one content directory output -folder under `Library/com.unity.addressables/aa`, and -`Library/BuildHistory`. Override any of them with `-BuildLayout`, -`-ContentDirectory`, or `-BuildHistory` — useful for auditing an older +```bash +# Linux / macOS +Skills/hybrid-duplication-audit/scripts/compare-hybrid-duplication.sh --project-root "" +``` + +By default either one locates everything it needs on its own: the +mirrored `Library/com.unity.addressables/buildlayout.json` +(Addressables keeps this in sync with its latest build), the one +content directory output folder under +`Library/com.unity.addressables/aa`, and `Library/BuildHistory`. +Override any of them with `-BuildLayout`/`--build-layout`, +`-ContentDirectory`/`--content-directory`, or +`-BuildHistory`/`--build-history` — useful for auditing an older build, or when more than one platform has been built (the script -requires `-ContentDirectory` explicitly in that case, since it can't -guess which platform you mean). +requires the content-directory override explicitly in that case, +since it can't guess which platform you mean). -**When overriding, make sure `-BuildLayout` and `-ContentDirectory` +**When overriding, make sure the build layout and content directory came from the same build.** Addressables can rebuild only some groups at a time, so the two can legitimately drift out of sync — nothing ties an Addressables build layout to a specific content-directory @@ -79,9 +90,12 @@ directory (or vice versa) won't error; it'll just report a wrong, usually much larger, "duplicate" total. The defaults are always safe in this respect, since both point at whatever is currently on disk. -Other flags: `-ToolPath` (if `UnityDataTool` isn't on PATH or set via -`UNITYDATATOOL_PATH`), `-KeepDatabase` (preserve the generated -database for follow-up queries), `-MaxRows`. +Other flags (PowerShell / bash): `-ToolPath`/`--tool-path` (if +`UnityDataTool` isn't on PATH or set via `UNITYDATATOOL_PATH` — on +Linux/macOS the built executable has no extension, e.g. +`UnityDataTool/bin/Release/net9.0/UnityDataTool`), +`-KeepDatabase`/`--keep-database` (preserve the generated database for +follow-up queries), `-MaxRows`/`--max-rows`. ## Reading the output diff --git a/Skills/hybrid-duplication-audit/scripts/compare-hybrid-duplication.sh b/Skills/hybrid-duplication-audit/scripts/compare-hybrid-duplication.sh new file mode 100755 index 0000000..a885ccd --- /dev/null +++ b/Skills/hybrid-duplication-audit/scripts/compare-hybrid-duplication.sh @@ -0,0 +1,248 @@ +#!/usr/bin/env bash +# +# Finds source assets duplicated across the AssetBundle / ContentDirectory boundary of a +# hybrid Addressables 4.x build (some groups build to .bundle files, others build through +# BuildPipeline.BuildContentDirectory / ContentDirectoryGroupSchema). +# +# Linux/macOS counterpart of Compare-HybridDuplication.ps1 -- same behavior, same queries. +# Requires UnityDataTool to be built (see ../../../README.md, "How to Build") and sqlite3 on +# PATH, exactly like the PowerShell version. +# +# See ../SKILL.md for the concept, the query this script runs, and how to read the output. +# +# DISCLAIMER: +# This script is provided "as-is," without any warranty of any kind, express or implied. +# By using this script, you agree that you understand its purpose and that you use it entirely +# at your own risk. Always review and test this script in a safe environment before applying it +# to a production system. + +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +project_root="." +tool_path="" +build_layout="" +content_directory="" +build_history="" +database="" +keep_database=0 +max_rows=50 + +usage() { + cat <<'EOF' +Usage: compare-hybrid-duplication.sh [options] + + --project-root Root of the Unity project to audit (default: .) + --tool-path Path to the UnityDataTool executable + --build-layout Addressables build layout report to read + (default: Library/com.unity.addressables/buildlayout.json) + --content-directory Content directory build output folder + (default: auto-detected under Library/com.unity.addressables/aa) + --build-history Project's build history folder + (default: Library/BuildHistory) + --database Output database path (default: a temp file, deleted afterwards) + --keep-database Keep the generated database instead of deleting it + --max-rows Maximum number of duplicate rows to print (default: 50) + -h, --help Show this help +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --project-root) project_root="$2"; shift 2 ;; + --tool-path) tool_path="$2"; shift 2 ;; + --build-layout) build_layout="$2"; shift 2 ;; + --content-directory) content_directory="$2"; shift 2 ;; + --build-history) build_history="$2"; shift 2 ;; + --database) database="$2"; shift 2 ;; + --keep-database) keep_database=1; shift ;; + --max-rows) max_rows="$2"; shift 2 ;; + -h|--help) usage; exit 0 ;; + *) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;; + esac +done + +add_commas() { + printf "%s" "$1" | sed -E ':a;s/(^[-]?[0-9]+)([0-9]{3})/\1,\2/;ta' +} + +# Resolve the UnityDataTool executable: explicit flag -> UNITYDATATOOL_PATH -> PATH -> this +# checkout's own build output (only valid while the skill is still inside a UnityDataTools +# clone; once copied into another project's .claude/skills, pass --tool-path or set +# UNITYDATATOOL_PATH instead). +if [[ -z "$tool_path" ]]; then + if [[ -n "${UNITYDATATOOL_PATH:-}" ]]; then + tool_path="$UNITYDATATOOL_PATH" + elif command -v UnityDataTool >/dev/null 2>&1; then + tool_path="$(command -v UnityDataTool)" + elif [[ -x "$script_dir/../../../UnityDataTool/bin/Release/net9.0/UnityDataTool" ]]; then + tool_path="$script_dir/../../../UnityDataTool/bin/Release/net9.0/UnityDataTool" + fi +fi + +if [[ -z "$tool_path" || ! -x "$tool_path" ]]; then + echo "Error: UnityDataTool executable not found. Tried: --tool-path, UNITYDATATOOL_PATH, PATH, and this checkout's own build output. Build it with 'dotnet build -c Release' or pass --tool-path explicitly." >&2 + exit 1 +fi + +if [[ ! -d "$project_root" ]]; then + echo "Error: Project root '$project_root' not found." >&2 + exit 1 +fi +project_root="$(cd "$project_root" && pwd)" + +if [[ -z "$build_layout" ]]; then + build_layout="$project_root/Library/com.unity.addressables/buildlayout.json" +fi +if [[ ! -f "$build_layout" ]]; then + echo "Error: Addressables build layout report not found at '$build_layout'. Build the project with Addressables first, or pass --build-layout explicitly." >&2 + exit 1 +fi + +if [[ -z "$content_directory" ]]; then + candidates=() + if [[ -d "$project_root/Library/com.unity.addressables/aa" ]]; then + while IFS= read -r -d '' manifest_hash_file; do + candidates+=("$(dirname "$manifest_hash_file")") + done < <(find "$project_root/Library/com.unity.addressables/aa" -name "BuildManifestHash.txt" -print0 2>/dev/null) + fi + + if [[ ${#candidates[@]} -eq 0 ]]; then + echo "Error: No content directory build output found under Library/com.unity.addressables/aa. Pass --content-directory explicitly, or this project may not have a ContentDirectoryGroupSchema group." >&2 + exit 1 + fi + if [[ ${#candidates[@]} -gt 1 ]]; then + echo "Error: More than one content directory build output found (multiple platforms built?): ${candidates[*]}. Pass --content-directory to pick one." >&2 + exit 1 + fi + content_directory="${candidates[0]}" +fi +if [[ ! -d "$content_directory" ]]; then + echo "Error: Content directory build output '$content_directory' not found." >&2 + exit 1 +fi + +if [[ -z "$build_history" ]]; then + build_history="$project_root/Library/BuildHistory" +fi +if [[ ! -d "$build_history" ]]; then + echo "Error: Build history folder '$build_history' not found. It is required to pair the content directory output with its ContentLayout.json." >&2 + exit 1 +fi + +delete_database_after=0 +if [[ -z "$database" ]]; then + database="$(mktemp -u "${TMPDIR:-/tmp}/hybrid-duplication-XXXXXX.db")" + if [[ "$keep_database" -eq 0 ]]; then + delete_database_after=1 + fi +fi +cleanup() { + if [[ "$delete_database_after" -eq 1 ]]; then + rm -f "$database" + fi +} +trap cleanup EXIT + +echo "Analyzing:" +echo " Build layout: $build_layout" +echo " Content directory: $content_directory" +echo " Build history: $build_history" +echo "" + +"$tool_path" analyze "$build_layout" "$content_directory" --build-history "$build_history" -o "$database" + +# Hybrid check: both a bundle-producing group and a content-directory build must be present, +# otherwise the intersection below is meaningless. +bundle_count="$(sqlite3 "$database" "SELECT COUNT(*) FROM addressables_build_bundles;")" +has_layout_table="$(sqlite3 "$database" "SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='content_layout_source_assets';")" +source_asset_count=0 +if [[ "$has_layout_table" -gt 0 ]]; then + source_asset_count="$(sqlite3 "$database" "SELECT COUNT(*) FROM content_layout_source_assets;")" +fi + +if [[ "$bundle_count" -eq 0 || "$source_asset_count" -eq 0 ]]; then + echo "This is not a hybrid build:" + echo " AssetBundle groups in the build layout: $bundle_count" + echo " Source assets in the content directory: $source_asset_count" + echo "" + echo "Both must be non-zero to audit cross-boundary duplication." + exit 2 +fi + +# The one query that answers "which source assets were built into both forms": bundle-side +# asset paths (explicit + implicit) intersected with the content directory's source assets. +# Matching by asset_path, not by object CRC, is what makes this catch cases a plain CRC diff +# (e.g. view_potential_duplicates, see Documentation/analyzer.md) would miss -- the same +# source asset can be built with a different variant set on each side, so the two copies do +# not have the same CRC despite being duplicates of the same asset. +duplicates_query=" +WITH bundle_assets AS ( + SELECT asset_path, serialized_size + streamed_size AS bytes + FROM addressables_build_explicit_assets + UNION ALL + SELECT asset_path, serialized_size + streamed_size AS bytes + FROM addressables_build_data_from_other_assets +), +content_dir_assets AS ( + SELECT DISTINCT asset_path FROM content_layout_source_assets +) +SELECT b.asset_path, SUM(b.bytes) AS bundle_bytes, COUNT(*) AS instances +FROM bundle_assets b +WHERE b.asset_path IN (SELECT asset_path FROM content_dir_assets) +GROUP BY b.asset_path +ORDER BY bundle_bytes DESC +LIMIT $max_rows; +" + +totals_query=" +WITH bundle_assets AS ( + SELECT serialized_size + streamed_size AS bytes FROM addressables_build_explicit_assets + UNION ALL + SELECT serialized_size + streamed_size AS bytes FROM addressables_build_data_from_other_assets +) +SELECT SUM(bytes) FROM bundle_assets; +" + +content_dir_total_query="SELECT SUM(size) FROM content_layout_binary_artifacts WHERE category != 'manifest';" + +duplicate_bytes_query=" +WITH bundle_assets AS ( + SELECT asset_path, serialized_size + streamed_size AS bytes FROM addressables_build_explicit_assets + UNION ALL + SELECT asset_path, serialized_size + streamed_size AS bytes FROM addressables_build_data_from_other_assets +) +SELECT COALESCE(SUM(bytes), 0) +FROM bundle_assets +WHERE asset_path IN (SELECT DISTINCT asset_path FROM content_layout_source_assets); +" + +bundle_total_bytes="$(sqlite3 "$database" "$totals_query")" +content_dir_total_bytes="$(sqlite3 "$database" "$content_dir_total_query")" +duplicate_bytes="$(sqlite3 "$database" "$duplicate_bytes_query")" + +declared_duplicate_count="$(grep -o '"DuplicatedAssetCount"[[:space:]]*:[[:space:]]*-\{0,1\}[0-9]*' "$build_layout" | grep -o -- '-\{0,1\}[0-9]*$' || true)" + +echo "=== Cross-boundary duplicates (built into both a bundle and the content directory) ===" +sqlite3 "$database" ".mode column" ".headers on" "$duplicates_query" +echo "" + +echo "=== Summary ===" +printf "Bundle-side asset payload: %15s bytes\n" "$(add_commas "$bundle_total_bytes")" +printf "Content directory payload: %15s bytes\n" "$(add_commas "$content_dir_total_bytes")" +printf "Cross-boundary duplicated: %15s bytes\n" "$(add_commas "$duplicate_bytes")" +if [[ "$bundle_total_bytes" -gt 0 ]]; then + pct="$(awk -v d="$duplicate_bytes" -v t="$bundle_total_bytes" 'BEGIN { printf "%.1f", (100.0 * d / t) }')" + echo " = ${pct}% of the bundle-side payload" +fi +if [[ -n "$declared_duplicate_count" ]]; then + echo "" + echo "Addressables' own DuplicatedAssetCount for this build: $declared_duplicate_count" + echo "(That count only scans AssetBundle-to-AssetBundle duplication -- it cannot see a copy that lives in the content directory, so it will read low or zero even when the total above is not.)" +fi + +if [[ "$delete_database_after" -eq 0 ]]; then + echo "" + echo "Database kept at: $database" +fi