[kasnder]

What

getAllApps selected apps.*, and getAllAppsForSignature kept the result in allAppsMemo for as long as the process ran. That kept two things in memory for every analysed app:

• the full analysis (raw trackerscan output, Info.plists, per-match class lists);
• both App Store records (the queue snapshot and the current storefront row, about 11 KB each).

buildSiteData and buildReverseIndex only read analysis.success and analysis.trackers.

• models/Apps.js: select only appid, analysed, details, the storefront details and {success, trackers}. The projection happens in SQL, so the rest of each analysis never reaches Node. Analyses that are not JSON objects (SQL NULL, JSON null, arrays) pass through unchanged, so the builders treat them exactly as before.
• routes/index.js: keep only the fields the builders read, and drop the rows once the cached views are built. Concurrent rebuilds for the same signature still share one load, through an in-flight promise.

The cached output is unchanged, so SCHEMA_VERSION does not need a bump.

The polling back-off that briefly sat on this branch was moved to its own PR, #38. The revert commit here leaves the branch identical to the memory change alone.

Testing

• Synthetic benchmark with about 2,820 apps, roughly the production count. The App Store records are real iTunes lookups, and the analyses are shaped like trackerscan_to_analysis.js output. Each variant ran in a separate process:

  Analysis size per app	Before: kept in memory	After	Before: process RSS	After
  Small (14 KB)	183 MB	2 MB	311 MB	209 MB
  Typical (40 KB)	317 MB	2 MB	454 MB	235 MB

  The reverse index and aggregate statistics were identical between the two versions in each run.

• Ran the new query against a local Postgres 16 with an ordinary row and with SQL NULL, JSON null, {"success":false} and array analyses. Each returned what the old code expected.

• npm test: 164/164 pass.

CI will fail at the npm audit step until #37 merges; the advisories are unrelated to this change.

https://claude.ai/code/session_01Fi3H98SycMK13rrr8ySSU1

[kasnder]

CI is red at the npm audit step only. npm test passes all 165 tests, including the new test/analyserHeartbeat.test.js.

The failure is not from this PR: it doesn't touch package.json or package-lock.json. The advisories it reports were published after main last went green on 23 September, so the same step would fail on main now:

• proxy-addr ≤ 2.0.7 (critical, GHSA-jqcg-44mw-7w3h), pulled in by Express. npm audit fix resolves it.
• brace-expansion 4.0.0–5.0.11 (high). npm audit fix resolves it.
• braces (all versions, high), reached only through the dev dependency nodemon → chokidar. npm audit offers only a breaking downgrade to nodemon@1.14.10, so it has no proper fix.

I couldn't find an existing fix to port, and I haven't regenerated the lockfile here. A re-run would fail the same way, so I haven't triggered one. Proposed fix, as a separate PR:

1. Run npm audit fix and commit the updated package-lock.json (fixes proxy-addr and brace-expansion).
2. Change the CI step to npm audit --omit=dev, so the dev-only braces advisory stops blocking. Alternatively, drop nodemon in favour of node --watch.

---

Generated by Claude Code