From b356e312c7a827cb42b13314edd8f18b995c041d Mon Sep 17 00:00:00 2001 From: Georges-Antoine Assi Date: Fri, 11 Sep 2026 16:03:59 -0400 Subject: [PATCH 01/30] docs(reference): regenerate the reference snippets for 5.3 Pins `sources.toml` at the 5.3 branch point so the environment-variable, scheduled-task and platform tables carry the release's contents. `gen_scheduled_tasks.py` read the task list out of `startup.py`'s `.init()` calls and `endpoints/tasks.py`'s `manual_tasks`. Upstream #4274 moved scheduling to RQ, and both now live in one `backend/tasks/registry.py` as the `SCHEDULED_TASKS` and `MANUAL_TASKS` dicts, so the parser reads those instead and keeps the registries' own order. The table picks up the recommendations index and the missing firmware cleanup. `UPC_LOOKUP_*` and the recommendation task's two variables are not in upstream's `env.template` yet (rommapp/romm#4467), so the two tables here were generated against that branch. They regenerate identically once it lands. Co-Authored-By: Claude Opus 5 (1M context) --- docs/resources/snippets/env-vars.md | 130 ++++++++++-------- docs/resources/snippets/scheduled-tasks.md | 14 +- .../resources/snippets/supported-platforms.md | 1 + scripts/gen_scheduled_tasks.py | 70 ++++------ scripts/sources.toml | 2 +- 5 files changed, 114 insertions(+), 103 deletions(-) diff --git a/docs/resources/snippets/env-vars.md b/docs/resources/snippets/env-vars.md index 9abdc432..e2864883 100644 --- a/docs/resources/snippets/env-vars.md +++ b/docs/resources/snippets/env-vars.md @@ -12,16 +12,17 @@ ### Database -| Variable | Default | Required | Description | -| ---------------- | --------- | :------: | ------------------------------------------------------------------------ | -| `ROMM_DB_DRIVER` | `mariadb` | | Database driver to use (mariadb, mysql, postgresql) | -| `DB_HOST` | | `✓` | Host name of the database instance | -| `DB_PORT` | `3306` | | Port number of the database instance | -| `DB_NAME` | `romm` | | Database name (should match MYSQL_DATABASE in MariaDB) | -| `DB_USER` | | `✓` | Database username (should match MARIADB_USER in MariaDB) | -| `DB_PASSWD` | | `✓` | Database password (should match MARIADB_PASSWORD in MariaDB) | -| `DB_ROOT_PASSWD` | | | Database root user password (only used by the bundled MariaDB container) | -| `DB_QUERY_JSON` | | | Extra query parameters for the database connection, as JSON | +| Variable | Default | Required | Description | +| ------------------------- | --------- | :------: | ----------------------------------------------------------------------------------------------------------- | +| `ROMM_DB_DRIVER` | `mariadb` | | Database driver to use (mariadb, mysql, postgresql) | +| `DB_HOST` | | `✓` | Host name of the database instance | +| `DB_PORT` | `3306` | | Port number of the database instance | +| `DB_NAME` | `romm` | | Database name (should match MYSQL_DATABASE in MariaDB) | +| `DB_USER` | | `✓` | Database username (should match MARIADB_USER in MariaDB) | +| `DB_PASSWD` | | `✓` | Database password (should match MARIADB_PASSWORD in MariaDB) | +| `DB_ROOT_PASSWD` | | | Database root user password (only used by the bundled MariaDB container) | +| `DB_QUERY_JSON` | | | Extra query parameters for the database connection, as JSON | +| `DB_POOL_RECYCLE_SECONDS` | `300` | | Retire a pooled connection after this long, before the server drops it for being idle (-1 to never recycle) | ### Redis/Valkey @@ -76,46 +77,60 @@ ### Metadata Providers -| Variable | Default | Required | Description | -| -------------------------------------- | ------- | :------: | --------------------------------------------------------- | -| `IGDB_CLIENT_ID` | | | Client ID for the IGDB API | -| `IGDB_CLIENT_SECRET` | | | Client secret for the IGDB API | -| `MOBYGAMES_API_KEY` | | | MobyGames secret API key | -| `SCREENSCRAPER_USER` | | | Screenscraper username | -| `SCREENSCRAPER_PASSWORD` | | | Screenscraper password | -| `STEAMGRIDDB_API_KEY` | | | SteamGridDB secret API key | -| `RETROACHIEVEMENTS_API_KEY` | | | RetroAchievements secret API key | -| `REFRESH_RETROACHIEVEMENTS_CACHE_DAYS` | `30` | | RetroAchievements metadata cache refresh interval in days | -| `PLAYMATCH_API_ENABLED` | `false` | | Enable PlayMatch API integration | -| `LAUNCHBOX_API_ENABLED` | `false` | | Enable LaunchBox API integration | -| `HASHEOUS_API_ENABLED` | `false` | | Enable Hasheous API integration | -| `FLASHPOINT_API_ENABLED` | `false` | | Enable Flashpoint API integration | -| `HLTB_API_ENABLED` | `false` | | Enable HowLongToBeat API integration | -| `TGDB_API_ENABLED` | `false` | | Enable TheGamesDB API integration | +| Variable | Default | Required | Description | +| -------------------------------------- | ------- | :------: | --------------------------------------------------------------------------------- | +| `IGDB_CLIENT_ID` | | | Client ID for the IGDB API | +| `IGDB_CLIENT_SECRET` | | | Client secret for the IGDB API | +| `MOBYGAMES_API_KEY` | | | MobyGames secret API key | +| `SCREENSCRAPER_USER` | | | Screenscraper username | +| `SCREENSCRAPER_PASSWORD` | | | Screenscraper password | +| `STEAMGRIDDB_API_KEY` | | | SteamGridDB secret API key | +| `RETROACHIEVEMENTS_API_KEY` | | | RetroAchievements secret API key | +| `REFRESH_RETROACHIEVEMENTS_CACHE_DAYS` | `30` | | RetroAchievements metadata cache refresh interval in days | +| `PLAYMATCH_API_ENABLED` | `false` | | Enable PlayMatch API integration | +| `LAUNCHBOX_API_ENABLED` | `false` | | Enable LaunchBox API integration | +| `HASHEOUS_API_ENABLED` | `false` | | Enable Hasheous API integration | +| `FLASHPOINT_API_ENABLED` | `false` | | Enable Flashpoint API integration | +| `HLTB_API_ENABLED` | `false` | | Enable HowLongToBeat API integration | +| `DEMOZOO_API_ENABLED` | `false` | | Enable Demozoo (filename tags (demozoo-N) / paste ID; no API key) | +| `POUET_API_ENABLED` | `false` | | Enable Pouët (filename tags (pouet-N) / paste ID; no API key) | +| `CSDB_API_ENABLED` | `false` | | Enable CSDb (filename tags (csdb-N) / paste ID / Demozoo CsdbRelease; no API key) | +| `STEAM_API_ENABLED` | `false` | | Enable Steam API integration (PC platforms only) | +| `TGDB_API_ENABLED` | `false` | | Enable TheGamesDB API integration | + +### Physical Games + +| Variable | Default | Required | Description | +| -------------------- | --------------------------------------------- | :------: | ------------------------------------------------------ | +| `UPC_LOOKUP_ENABLED` | `true` | | Look a barcode up by UPC when adding a physical game | +| `UPC_LOOKUP_API_KEY` | | | Key for the UPC lookup service, if your plan needs one | +| `UPC_LOOKUP_URL` | `https://api.upcitemdb.com/prod/trial/lookup` | | UPC lookup endpoint | ### Scans & Tasks -| Variable | Default | Required | Description | -| -------------------------------------------------- | ----------- | :------: | ----------------------------------------------------------------------------------------- | -| `SCAN_TIMEOUT` | `14400` | | Timeout for background scan/rescan tasks in seconds | -| `SCAN_WORKERS` | `4` | | How many ROMs a scan processes at once | -| `TASK_TIMEOUT` | `300` | | Timeout for other background tasks in seconds | -| `TASK_RESULT_TTL` | `86400` | | How long to keep task results in Valkey in seconds | -| `SEVEN_ZIP_TIMEOUT` | `60` | | Timeout for 7-Zip operations in seconds | -| `ENABLE_RESCAN_ON_FILESYSTEM_CHANGE` | `false` | | Re-scan the library automatically when the filesystem changes | -| `RESCAN_ON_FILESYSTEM_CHANGE_DELAY` | `5` | | Delay in minutes before re-scanning after a filesystem change | -| `ENABLE_SCHEDULED_RESCAN` | `false` | | Enable scheduled library re-scans | -| `SCHEDULED_RESCAN_CRON` | `0 3 * * *` | | Cron expression for scheduled re-scans | -| `ENABLE_SCHEDULED_UPDATE_SWITCH_TITLEDB` | `false` | | Enable scheduled Switch TitleDB index updates | -| `SCHEDULED_UPDATE_SWITCH_TITLEDB_CRON` | `0 4 * * *` | | Cron expression for scheduled Switch TitleDB updates | -| `ENABLE_SCHEDULED_UPDATE_LAUNCHBOX_METADATA` | `false` | | Enable scheduled LaunchBox metadata updates | -| `SCHEDULED_UPDATE_LAUNCHBOX_METADATA_CRON` | `0 4 * * *` | | Cron expression for scheduled LaunchBox metadata updates | -| `ENABLE_SCHEDULED_CONVERT_IMAGES_TO_WEBP` | `false` | | Enable scheduled conversion of images to WebP | -| `SCHEDULED_CONVERT_IMAGES_TO_WEBP_CRON` | `0 4 * * *` | | Cron expression for scheduled WebP conversion | -| `ENABLE_SCHEDULED_CLEANUP_ORPHANED_RESOURCES` | `false` | | Enable scheduled cleanup of orphaned resources (covers, screenshots) left by deleted ROMs | -| `SCHEDULED_CLEANUP_ORPHANED_RESOURCES_CRON` | `0 5 * * *` | | Cron expression for scheduled orphaned resource cleanup | -| `ENABLE_SCHEDULED_RETROACHIEVEMENTS_PROGRESS_SYNC` | `false` | | Enable scheduled RetroAchievements progress sync | -| `SCHEDULED_RETROACHIEVEMENTS_PROGRESS_SYNC_CRON` | `0 4 * * *` | | Cron expression for scheduled RetroAchievements sync | +| Variable | Default | Required | Description | +| -------------------------------------------------- | ------------ | :------: | ----------------------------------------------------------------------------------------- | +| `SCAN_TIMEOUT` | `14400` | | Timeout for background scan/rescan tasks in seconds | +| `SCAN_WORKERS` | `4` | | How many ROMs a scan processes at once | +| `TASK_TIMEOUT` | `300` | | Timeout for other background tasks in seconds | +| `TASK_RESULT_TTL` | `86400` | | How long to keep task results in Valkey in seconds | +| `SEVEN_ZIP_TIMEOUT` | `60` | | Timeout for 7-Zip operations in seconds | +| `ENABLE_RESCAN_ON_FILESYSTEM_CHANGE` | `false` | | Re-scan the library automatically when the filesystem changes | +| `RESCAN_ON_FILESYSTEM_CHANGE_DELAY` | `5` | | Delay in minutes before re-scanning after a filesystem change | +| `ENABLE_SCHEDULED_RESCAN` | `false` | | Enable scheduled library re-scans | +| `SCHEDULED_RESCAN_CRON` | `0 3 * * *` | | Cron expression for scheduled re-scans | +| `ENABLE_SCHEDULED_UPDATE_SWITCH_TITLEDB` | `false` | | Enable scheduled Switch TitleDB index updates | +| `SCHEDULED_UPDATE_SWITCH_TITLEDB_CRON` | `0 4 * * *` | | Cron expression for scheduled Switch TitleDB updates | +| `ENABLE_SCHEDULED_UPDATE_LAUNCHBOX_METADATA` | `false` | | Enable scheduled LaunchBox metadata updates | +| `SCHEDULED_UPDATE_LAUNCHBOX_METADATA_CRON` | `0 4 * * *` | | Cron expression for scheduled LaunchBox metadata updates | +| `ENABLE_SCHEDULED_CONVERT_IMAGES_TO_WEBP` | `false` | | Enable scheduled conversion of images to WebP | +| `SCHEDULED_CONVERT_IMAGES_TO_WEBP_CRON` | `0 4 * * *` | | Cron expression for scheduled WebP conversion | +| `ENABLE_SCHEDULED_CLEANUP_ORPHANED_RESOURCES` | `false` | | Enable scheduled cleanup of orphaned resources (covers, screenshots) left by deleted ROMs | +| `SCHEDULED_CLEANUP_ORPHANED_RESOURCES_CRON` | `0 5 * * *` | | Cron expression for scheduled orphaned resource cleanup | +| `ENABLE_SCHEDULED_RETROACHIEVEMENTS_PROGRESS_SYNC` | `false` | | Enable scheduled RetroAchievements progress sync | +| `SCHEDULED_RETROACHIEVEMENTS_PROGRESS_SYNC_CRON` | `0 4 * * *` | | Cron expression for scheduled RetroAchievements sync | +| `ENABLE_SCHEDULED_BUILD_RECOMMENDATIONS` | `true` | | Enable the scheduled rebuild of the recommendations index | +| `SCHEDULED_BUILD_RECOMMENDATIONS_CRON` | `30 5 * * *` | | Cron expression for the recommendations index rebuild | ### Sync @@ -130,11 +145,12 @@ ### Emulation -| Variable | Default | Required | Description | -| --------------------- | ------- | :------: | ----------------------------------------------------------- | -| `DISABLE_EMULATOR_JS` | `false` | | Disable in-browser play via EmulatorJS | -| `DISABLE_RUFFLE_RS` | `false` | | Disable in-browser Flash playback via RuffleRS | -| `DISABLE_JSDOS` | `false` | | Disable in-browser Windows 3.x and 9x playback via `js-dos` | +| Variable | Default | Required | Description | +| --------------------- | ------- | :------: | -------------------------------------------------------- | +| `DISABLE_EMULATOR_JS` | `false` | | Disable in-browser play via EmulatorJS | +| `DISABLE_RUFFLE_RS` | `false` | | Disable in-browser Flash playback via RuffleRS | +| `DISABLE_JSDOS` | `false` | | Disable in-browser Win3.x and Win9.x playback via js-dos | +| `DISABLE_PICO8` | `false` | | Disable in-browser PICO-8 playback via FAKE-08 | ### Integrations @@ -202,7 +218,9 @@ ### Emulator Streaming -| Variable | Default | Required | Description | -| ------------------------- | ------- | :------: | --------------------------------------------------------------------------------- | -| `STREAMING_BROKER_SECRET` | | | - | -| `STREAMING_SAVE_TIMEOUT` | `45` | | Seconds to wait for a broker save-and-exit (raise if a broker has SAVE_WAIT > 45) | +| Variable | Default | Required | Description | +| ------------------------------- | ------- | :------: | -------------------------------------------------------------------------------------------------- | +| `STREAMING_BROKER_SECRET` | | | Shared secret matching streaming containers' BROKER_SECRET, required for broker auth | +| `STREAMING_SAVE_TIMEOUT` | `45` | | Seconds to wait for a broker save-and-exit (raise if a broker has SAVE_WAIT > 45) | +| `STREAMING_LAUNCH_TIMEOUT` | `600` | | Seconds a webstation activate may take, covering pkg/archive extraction before the emulator starts | +| `STREAMING_STATE_HISTORY_LIMIT` | `50` | | Save states kept per ROM, emulator and user; oldest are pruned past this (0 disables) | diff --git a/docs/resources/snippets/scheduled-tasks.md b/docs/resources/snippets/scheduled-tasks.md index 79c86ae9..b920895a 100644 --- a/docs/resources/snippets/scheduled-tasks.md +++ b/docs/resources/snippets/scheduled-tasks.md @@ -2,18 +2,20 @@ | Task | Type | Default schedule | Enable var | Schedule/delay var | Purpose | | ----------------------------------------- | --------- | ---------------- | -------------------------------------------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| Scheduled netplay cleanup | Scheduled | `*/30 * * * *` | `-` | `-` | Cleans up empty netplay rooms. Always on, not configurable. | -| Scheduled ZIP cache cleanup | Scheduled | `0 4 * * *` | `-` | `-` | Removes stale cached ZIP files based on tiered TTL. Always on, not configurable. | -| Scheduled upload tmp cleanup | Scheduled | `0 * * * *` | `-` | `-` | Cleans up orphaned chunked-upload temp directories. Always on, not configurable. | -| Cleanup orphaned resources | Scheduled | `0 5 * * *` | `ENABLE_SCHEDULED_CLEANUP_ORPHANED_RESOURCES` | `SCHEDULED_CLEANUP_ORPHANED_RESOURCES_CRON` | Clean up orphaned resources in the ROMs directory. | | Scheduled rescan | Scheduled | `0 3 * * *` | `ENABLE_SCHEDULED_RESCAN` | `SCHEDULED_RESCAN_CRON` | Rescans the entire library. | -| Scheduled Switch TitleDB update | Scheduled | `0 4 * * *` | `ENABLE_SCHEDULED_UPDATE_SWITCH_TITLEDB` | `SCHEDULED_UPDATE_SWITCH_TITLEDB_CRON` | Updates the Nintendo Switch TitleDB file. | | Scheduled LaunchBox metadata update | Scheduled | `0 4 * * *` | `ENABLE_SCHEDULED_UPDATE_LAUNCHBOX_METADATA` | `SCHEDULED_UPDATE_LAUNCHBOX_METADATA_CRON` | Updates the LaunchBox metadata store. | +| Scheduled Switch TitleDB update | Scheduled | `0 4 * * *` | `ENABLE_SCHEDULED_UPDATE_SWITCH_TITLEDB` | `SCHEDULED_UPDATE_SWITCH_TITLEDB_CRON` | Updates the Nintendo Switch TitleDB file. | +| Build recommendations index | Scheduled | `30 5 * * *` | `ENABLE_SCHEDULED_BUILD_RECOMMENDATIONS` | `SCHEDULED_BUILD_RECOMMENDATIONS_CRON` | Rebuilds the similar-games index from library metadata, play history and collections. | | Convert images to WebP | Scheduled | `0 4 * * *` | `ENABLE_SCHEDULED_CONVERT_IMAGES_TO_WEBP` | `SCHEDULED_CONVERT_IMAGES_TO_WEBP_CRON` | Convert existing image files (PNG, JPG, BMP, TIFF, GIF) to WebP format for better performance. | +| Scheduled ZIP cache cleanup | Scheduled | `0 4 * * *` | `-` | `-` | Removes stale cached ZIP files based on tiered TTL. Always on, not configurable. | +| Cleanup orphaned resources | Scheduled | `0 5 * * *` | `ENABLE_SCHEDULED_CLEANUP_ORPHANED_RESOURCES` | `SCHEDULED_CLEANUP_ORPHANED_RESOURCES_CRON` | Clean up orphaned resources in the ROMs directory. | +| Scheduled netplay cleanup | Scheduled | `*/30 * * * *` | `-` | `-` | Cleans up empty netplay rooms. Always on, not configurable. | +| Scheduled upload tmp cleanup | Scheduled | `0 * * * *` | `-` | `-` | Cleans up orphaned chunked-upload temp directories. Always on, not configurable. | | Scheduled RetroAchievements progress sync | Scheduled | `0 4 * * *` | `ENABLE_SCHEDULED_RETROACHIEVEMENTS_PROGRESS_SYNC` | `SCHEDULED_RETROACHIEVEMENTS_PROGRESS_SYNC_CRON` | Updates RetroAchievements progress for all users. | | Push-Pull Sync | Scheduled | `*/30 * * * *` | `ENABLE_SYNC_PUSH_PULL` | `SYNC_PUSH_PULL_CRON` | Sync saves with devices via SSH/SFTP. | | Cleanup missing ROMs | Manual | `-` | `-` | `-` | Delete all ROMs flagged as missing from the filesystem from the database. | -| Recompute save content hashes | Manual | `-` | `-` | `-` | Re-scan every save row and rewrite content_hash with the current compute_content_hash algorithm. One-time recovery after the zip-hash dispatch fix. | +| Cleanup missing firmware | Manual | `-` | `-` | `-` | Delete all firmware flagged as missing from the filesystem from the database. | | Sync Folder Scan | Manual | `-` | `ENABLE_SYNC_FOLDER_WATCHER` | `-` | Scan device sync folders for new save files. | +| Recompute save content hashes | Manual | `-` | `-` | `-` | Re-scan every save row and rewrite content_hash with the current compute_content_hash algorithm. One-time recovery after the zip-hash dispatch fix. | | Filesystem watcher | Watcher | `-` | `ENABLE_RESCAN_ON_FILESYSTEM_CHANGE` | `RESCAN_ON_FILESYSTEM_CHANGE_DELAY` | Watch the library folder and trigger a rescan on changes. | | Sync folder watcher | Watcher | `-` | `ENABLE_SYNC_FOLDER_WATCHER` | `SYNC_FOLDER_SCAN_DELAY` | Watch the sync folder and trigger a scan on changes. | diff --git a/docs/resources/snippets/supported-platforms.md b/docs/resources/snippets/supported-platforms.md index 2b0e1b83..428e5d10 100644 --- a/docs/resources/snippets/supported-platforms.md +++ b/docs/resources/snippets/supported-platforms.md @@ -123,6 +123,7 @@ | Dreamcast | `dc` | igdb logo screenscraper logo mobygames logo launchbox logo hasheous logo retroachivements logo howlongtobeat logo libretro logo | | DVD Player | `dvd-player` | igdb logo mobygames logo howlongtobeat logo | | e-Reader / Card-e Reader | `e-reader-slash-card-e-reader` | igdb logo | +| EasyRPG | `rpg-maker` | screenscraper logo libretro logo | | ECD Micromind | `ecd-micromind` | mobygames logo | | EDSAC | `edsac` | igdb logo | | Elektor TV Games Computer | `elektor` | igdb logo retroachivements logo | diff --git a/scripts/gen_scheduled_tasks.py b/scripts/gen_scheduled_tasks.py index 1b8ef945..50f1bf6f 100644 --- a/scripts/gen_scheduled_tasks.py +++ b/scripts/gen_scheduled_tasks.py @@ -10,8 +10,7 @@ Sources, all fetched at the ref pinned in sources.toml: - backend/startup.py which tasks the scheduler calls init() on - backend/endpoints/tasks.py which tasks the Tasks page can run by hand + backend/tasks/registry.py the SCHEDULED_TASKS and MANUAL_TASKS registries backend/tasks/**.py each task's title, enabled flag and cron default env.template resolves env constants to documented defaults @@ -78,35 +77,16 @@ def import_map(tree: ast.Module) -> dict[str, str]: return out -def scheduled_module_paths(startup_src: str) -> list[str]: - """Module paths for every task startup.py calls `.init()` on, in order.""" - tree = ast.parse(startup_src) - imports = import_map(tree) - - paths: list[str] = [] - for node in ast.walk(tree): - if not isinstance(node, ast.Call): - continue - func = node.func - if not isinstance(func, ast.Attribute) or func.attr != "init": - continue - if not isinstance(func.value, ast.Name): - continue - path = imports.get(func.value.id) - if path and path not in paths: - paths.append(path) - - if not paths: - raise UpstreamDrift( - "no `.init()` calls found in backend/startup.py. The scheduler " - "entrypoint moved or changed shape, so this parser needs updating." - ) - return paths +def registry_module_paths(registry_src: str, name: str) -> list[str]: + """Module paths for the task singletons a registry dict maps to. + `SCHEDULED_TASKS = {"scan_library": scan_library_task, ...}` + -> ["backend/tasks/scheduled/scan_library.py", ...] -def manual_module_paths(endpoints_src: str) -> list[str]: - """Module paths for the entries of the `manual_tasks` registry list.""" - tree = ast.parse(endpoints_src) + Registry order is the table's order, so the dict is read in source order + rather than walked. + """ + tree = ast.parse(registry_src) imports = import_map(tree) for node in ast.walk(tree): @@ -116,21 +96,30 @@ def manual_module_paths(endpoints_src: str) -> list[str]: elif isinstance(node, ast.Assign) and len(node.targets) == 1: if isinstance(node.targets[0], ast.Name): target = node.targets[0].id - if target != "manual_tasks" or node.value is None: + if target != name or not isinstance(node.value, ast.Dict): continue - names = {n.id for n in ast.walk(node.value) if isinstance(n, ast.Name)} paths: list[str] = [] - for name in sorted(names): - path = imports.get(name) - if path and path not in paths: + for value in node.value.values: + if not isinstance(value, ast.Name): + raise UpstreamDrift( + f"{name} in backend/tasks/registry.py maps a key to a " + f"{type(value).__name__} rather than an imported task " + "singleton, which this parser cannot resolve." + ) + path = imports.get(value.id) + if path is None: + raise UpstreamDrift( + f"{name} references {value.id}, which is not imported from a " + "tasks.* module in backend/tasks/registry.py." + ) + if path not in paths: paths.append(path) - if paths: - return paths + return paths raise UpstreamDrift( - "no `manual_tasks` registry found in backend/endpoints/tasks.py. The " - "registry moved or was renamed, so this parser needs updating." + f"no `{name}` dict found in backend/tasks/registry.py. The registry moved " + "or changed shape, so this parser needs updating." ) @@ -214,8 +203,9 @@ def build_row(path: str, kind: str, env: dict[str, dict]) -> dict: def collect(env: dict[str, dict]) -> list[dict]: - scheduled = scheduled_module_paths(fetch_text(romm_raw_url("backend/startup.py"))) - manual = manual_module_paths(fetch_text(romm_raw_url("backend/endpoints/tasks.py"))) + registry_src = fetch_text(romm_raw_url("backend/tasks/registry.py")) + scheduled = registry_module_paths(registry_src, "SCHEDULED_TASKS") + manual = registry_module_paths(registry_src, "MANUAL_TASKS") rows = [build_row(p, "Scheduled", env) for p in scheduled] # A task in both registries is scheduled and also runnable by hand, so it is diff --git a/scripts/sources.toml b/scripts/sources.toml index f63c1bcf..9db33591 100644 --- a/scripts/sources.toml +++ b/scripts/sources.toml @@ -13,4 +13,4 @@ repo = "rommapp/romm" # Set to a tag (e.g. "v5.0.0") or a full SHA. Generators fetch raw files # from raw.githubusercontent.com/${repo}/${ref}/... -ref = "5.2.0" +ref = "1fdf0926658e20d6bf62f0f000e5bb33d7f933b5" From a742ec7893e986a8d90030cf0ec34b66787ee316 Mon Sep 17 00:00:00 2001 From: Georges-Antoine Assi Date: Fri, 11 Sep 2026 16:06:55 -0400 Subject: [PATCH 02/30] docs(streaming): rewrite for the one-entry-per-container config Streaming is now one webstation container serving every platform you point at it, so `streaming.containers` takes one entry per container with a `platforms` map, and the container's own keys are the defaults for every platform it serves. Both the guide and the schema reference still described one entry per emulator. Also covers what the release adds around that: container pools, the in-game save archive, the memory card library, live disc swap and the disc a state remounts, joinable sessions, admin desktop sessions and the container fleet, and `STREAMING_LAUNCH_TIMEOUT`. The supported platform table is rebuilt from the broker's capability tables, which now reach well past the original three emulators. Co-Authored-By: Claude Opus 5 (1M context) --- docs/reference/configuration-file.md | 59 +++++---- docs/using/emulator-streaming.md | 174 ++++++++++++++++++++------- 2 files changed, 169 insertions(+), 64 deletions(-) diff --git a/docs/reference/configuration-file.md b/docs/reference/configuration-file.md index 9f14216e..27d53405 100644 --- a/docs/reference/configuration-file.md +++ b/docs/reference/configuration-file.md @@ -489,36 +489,49 @@ streaming: ### `streaming.containers` -One entry per emulator container, where each entry maps a [platform slug](../platforms/supported-platforms.md) to the container that streams it. - -| Key | Required | Purpose | -| ------------------ | -------- | -------------------------------------------------------------------------------------------------------------- | -| `platform` | Yes | Platform slug this container serves (e.g. `ps2`, `ngc`, `wii`, `xbox`) | -| `host` | Yes | Browser-facing Selkies web UI (must be reachable from clients and served over **HTTPS**) | -| `broker_host` | No | Server-side broker API (derived from `host` if omitted) | -| `label` | Yes | Text shown on the play action (e.g. `PCSX2`) | -| `broker_secret` | No | Secret for this container, used only when the `STREAMING_BROKER_SECRET` env var is unset | -| `memory_card_sync` | No | Sync the whole memory card to the RomM library on `ps2` and `ngc` (GameCube), ignored on cardless platforms | -| `library_path` | No | In-container path to the RomM library if it is mounted somewhere other than the default `/romm/library` | -| `emulator` | No | Lowercased name grouping this container's states and memory cards, defaults to `label`, then the platform slug | - -See [Emulator Streaming → Memory cards](../using/emulator-streaming.md#memory-cards) for how `memory_card_sync` behaves. +**One entry per container**, not per platform. A container serves every platform listed in its `platforms` map, and its own keys are the defaults for all of them. + +| Key | Required | Purpose | +| ------------------ | -------- | --------------------------------------------------------------------------------------------------------------- | +| `host` | Yes | Browser-facing Selkies web UI, served over **HTTPS**, or a path when reverse proxied onto RomM's own origin | +| `platforms` | Yes | Map of [platform slug](../platforms/supported-platforms.md) to the emulator serving it, or to an override block | +| `protocol` | Yes | `webstation`. Omitted, the entry is read as a deprecated per-emulator broker mod | +| `label` | Yes | Name for the container, used on the play action for any platform that sets none of its own | +| `subfolder` | No | URL prefix the broker is served under, matching the container's `SUBFOLDER` | +| `broker_host` | No | Server-to-broker API base. Derived from `host` when omitted, and **required** when `host` is a path | +| `broker_secret` | No | Secret for this container, used only when the `STREAMING_BROKER_SECRET` env var is unset | +| `library_path` | No | In-container path to the RomM library, if it is mounted somewhere other than the default `/romm/library` | +| `emulator` | No | Lowercased name grouping this container's states and memory cards, defaults to `label` | +| `memory_card_sync` | No | Sync the whole memory card to the RomM library, honoured on `ps2` and `ngc` and ignored elsewhere | + +A `platforms` value is either the emulator name as a bare string, or a block overriding `emulator`, `label` and `memory_card_sync` for that one platform. ```yaml streaming: enabled: true containers: - - platform: ps2 - host: https://192.168.1.51:3001 # browser-facing, must be HTTPS - broker_host: http://192.168.1.51:8000 # server-to-container, HTTP ok - label: PCSX2 - memory_card_sync: true # keep the PS2 memory card in your RomM library - - platform: ngc # ngc/wii can share one Dolphin container - host: https://192.168.1.51:3002 - broker_host: http://192.168.1.51:8001 - label: Dolphin + - protocol: webstation + host: https://192.168.1.56:3010 # browser-facing, must be HTTPS + subfolder: /streaming # matches the container's SUBFOLDER + library_path: /romm # where it mounts your ROM library + broker_secret: change-me # matches the container's BROKER_SECRET + label: Emulation station + platforms: + snes: retroarch # the emulator name directly... + ps2: # ...or a block overriding container keys + emulator: pcsx2 + label: PCSX2 + memory_card_sync: true + ngc: + emulator: dolphin + label: Dolphin + memory_card_sync: true ``` +List a platform on several containers and they become a pool, where a claim takes the first free one. Pool members have to agree on `emulator`, `memory_card_sync` and `protocol`, and are told apart by their broker host, so give each a distinct `broker_host` (see [Emulator Streaming → How a session works](../using/emulator-streaming.md#how-a-session-works)). + +See [Emulator Streaming → Memory cards](../using/emulator-streaming.md#memory-cards) for how `memory_card_sync` behaves, and [Migrating to webstation](../using/emulator-streaming-migration.md) if you still run the per-emulator broker mods. + --- ## Related diff --git a/docs/using/emulator-streaming.md b/docs/using/emulator-streaming.md index 3e9c7e60..ac9c329a 100644 --- a/docs/using/emulator-streaming.md +++ b/docs/using/emulator-streaming.md @@ -5,45 +5,79 @@ description: Launch games into a native emulator running in a container # Emulator Streaming -Emulator streaming launches a game into a **native** emulator running in a separate container and streams the picture, sound, and input back to your browser. Unlike [in-browser play](in-browser-play/emulatorjs.md), the emulation runs server-side on real emulator binaries (PCSX2, Dolphin, xemu), so the heavy lifting happens on the host rather than in the client. The server claims a session, tells the emulator which ROM to launch, and shows the live stream inside a player view, with save-state and volume controls in the toolbar. +Emulator streaming launches a game into a **native** emulator running in a separate container and streams the picture, sound, and input back to your browser. Unlike [in-browser play](in-browser-play/emulatorjs.md), the emulation runs server-side on real emulator binaries (PCSX2, Dolphin, RPCS3, RetroArch and friends), so the heavy lifting happens on the host rather than in the client. RomM claims a container, tells its broker which ROM to launch, and shows the live stream inside a player view with save, state, volume and disc controls. -Each emulator runs in its own [linuxserver](https://docs.linuxserver.io) container with a [Selkies](https://github.com/selkies-project/selkies) WebRTC stream and a small HTTP broker sidecar that RomM talks to. Nothing appears in the UI until you configure at least one container. +Streaming runs on a [docker-webstation](https://github.com/linuxserver/docker-webstation) container: a [Selkies](https://github.com/selkies-project/selkies) WebRTC desktop with every supported emulator installed, plus the [romm-broker](https://github.com/romm-streaming/romm-broker) sidecar that RomM talks to. One container serves **every** platform you point at it. Nothing appears in the UI until you configure at least one. - -!!! info "Already running per-emulator containers?" - Streaming is moving to a single webstation container that serves every platform. See [Migrating to webstation](emulator-streaming-migration.md). +A platform RomM can both emulate in the browser and stream offers the two as separate actions, so browser play stays available on a platform you also stream. -!!! warning "Work in progress" - This is the first release of the streaming framework and ships with three emulators. More integrations (rpcs3 for PS3, and others) are planned as separate follow-ups. +!!! info "Coming from the per-emulator broker mods?" + A container without `protocol: webstation` still works, logs a startup warning, and will stop being supported in a future release. See [Migrating to webstation](emulator-streaming-migration.md) for the config rewrite. -## Supported emulators +## How a session works -Each emulator ships as a companion Docker mod repo with the broker sidecar and a worked `docker-compose.yml`. +A container drives one display, so it holds **one session at a time**, whatever platform that session is playing. Sessions live in [Valkey](../install/redis-or-valkey.md) behind an atomic claim, so several API workers stay consistent. A session is bound to the user who claimed it, and only that owner or an admin can control or release it. -| Platform slug | Emulator | Save states | Manual slots | Autosave slot | Broker repo | -| ------------- | -------- | ----------- | ------------ | ------------- | ------------------------------------------------------------------------------------- | -| `ps2` | PCSX2 | Yes | 9 | Slot 10 | [pcsx2-romm-integration](https://github.com/LoneAngelFayt/pcsx2-romm-integration) | -| `ngc`, `wii` | Dolphin | Yes | 7 | Slot 8 | [dolphin-romm-integration](https://github.com/LoneAngelFayt/dolphin-romm-integration) | -| `xbox` | xemu | Yes | 9 | Slot 10 | [xemu-romm-integration](https://github.com/LoneAngelFayt/xemu-romm-integration) | +Configure a platform on more than one container and those containers become a **pool**. A claim walks them in config order and takes the first free one, so two people can play the same platform at once as long as there is a container each. When every container is held, RomM looks for sessions whose heartbeat has gone stale (a closed tab, a crashed browser) and tears those down before reporting the platform busy. -The broker launches ROMs as direct files, so **archive extraction is not supported**. +Pool members have to agree on `emulator`, `memory_card_sync` and `protocol`. Those decide where saves are filed and which controls exist, so a player landing on either member has to find the same setup. Containers that disagree are treated as separate setups rather than a pool. + +## Supported platforms + +The broker ships standalone emulators for the platforms below, and RetroArch for everything else. + +| Platform slug | Emulator | Save states | Memory card | Disc swap | +| ------------------- | ----------- | --------------------------- | ----------- | --------- | +| `ps2` | PCSX2 | Slots 1-9, autosave slot 10 | Yes | Manual | +| `ngc` | Dolphin | Slots 1-7, autosave slot 8 | Yes | - | +| `wii` | Dolphin | Slots 1-7, autosave slot 8 | - | - | +| `psx` | DuckStation | One resume state | - | - | +| `ps3` | RPCS3 | One resume state | - | - | +| `xbox` | xemu | - | - | - | +| `xbox360` | Xenia | - | - | - | +| `wiiu` | Cemu | - | - | - | +| `switch` | Eden | - | - | - | +| `3ds` | Azahar | - | - | - | +| `ps4` | shadPS4 | - | - | - | +| `psp` | PPSSPP | - | - | - | +| `dc` | Flycast | - | - | Yes | +| _(anything else)_ | RetroArch | One resume state | - | Varies | +| _(adventure games)_ | ScummVM | One resume state | - | - | + +RetroArch serves dozens of platforms from one container and the broker picks the libretro core, so RomM labels the action with that core's name (`RA Snes9x`, `RA mGBA`). RomM never selects a core itself, that is the broker's `retroarch_platforms.json`. -Only **one session per platform** can be active at a time, since there is a single emulator container behind it. Sessions are stored in [Valkey](../install/redis-or-valkey.md) with an atomic claim, so multiple workers stay consistent. The session is bound to the user who claimed it, and only that owner or an admin can control or release it, though an admin can force-release a stuck session. +The broker launches ROMs as direct files, so **archive extraction is not supported**. ## Saves and save states -**Save states** are the emulator's own quick-save slots: numbered manual slots plus a dedicated autosave slot per platform (see the table above). The autosave slot is reserved for **Save & Exit** and is overwritten on the next exit. +Three different things travel between a container and your library, and which ones apply depends on the platform. + +### Save states + +**Save states** are the emulator's own snapshots. Platforms fall into three groups, per the table above: + +- **Numbered slots plus an autosave slot** (PCSX2, Dolphin). The autosave slot is reserved for save-and-exit and is overwritten on the next exit. +- **One resume state** (DuckStation, RPCS3, RetroArch, ScummVM). These emulators write a state as they terminate, so there is no slot grid, just the single state that save and resume both land in. +- **No states at all** (xemu, Xenia, Cemu, Eden, Azahar, shadPS4). Persistence is the game's own save data instead. + +Each state is copied off the container into your library as it is written, with a thumbnail grabbed from the stream canvas alongside it, and the state written by save-and-exit is filed when the session ends. Your stored states are pushed back down when you claim a container, so they roam across containers and survive a container rebuild. RomM keeps the newest states per game, emulator, and user, and prunes the rest past `STREAMING_STATE_HISTORY_LIMIT` (default `50`, `0` to keep everything). + +These streaming states are stored separately from RomM's [per-user saves and states](saves-and-states.md) from in-browser play, since the file formats do not interchange. -Each state is copied off the container into your library as it is written, with a thumbnail alongside it, and the state written by **Save & Exit** is filed when the session ends. Your stored states are offered the next time you launch that game, so you can carry on from one instead of booting fresh. RomM keeps the newest states per game, emulator, and user, and prunes the rest past `STREAMING_STATE_HISTORY_LIMIT` (default `50`, `0` to keep everything). +### In-game saves -These streaming states are stored separately from RomM's [per-user saves and states](saves-and-states.md) from in-browser play. +The emulator's **own** save data (NAND, battery saves, the emulated user profile) is pulled off the container as a single zip archive when a session ends, and stored as one save asset so the whole set travels as a unit. This is what persists progress on the platforms with no save states. -## Memory cards +### Memory cards -On platforms with a memory card (**PS2** and **GameCube**), you can opt a container into whole-card sync with `memory_card_sync: true`. The card then lives in your RomM library rather than on the container: RomM loads your card in when a session starts, copies it back when you exit, and leaves the container's slot blank in between. Your most recently used card for that emulator loads by default, and you can keep several cards and pick which one a session mounts. +On **PS2** and **GameCube**, you can opt a container into whole-card sync with `memory_card_sync: true`. The card then lives in your RomM library rather than on the container: RomM loads your card in when a session starts, copies it back when you exit, and leaves the container's slot blank in between. Whole-card sync **replaces** the in-game save path above for that container. -PCSX2 only serves its card when Slot 1 holds a **Folder** card rather than a **File** card. With a File card the broker refuses the transfer and the session will not start, so set the card type before turning the flag on (see the [PCSX2 broker's memory card setup](https://github.com/LoneAngelFayt/pcsx2-romm-integration#memory-card-setup)). Dolphin's broker pins its own folder card, so GameCube needs no extra setup. +Cards are a library of their own. You can keep several, name and rename them, snapshot the current state as a version and roll back to one, share a card with other users, and download it. Your most recently used card for that emulator loads by default. + +Setting `memory_card_sync` on a platform RomM knows has no card (Wii, xemu) is ignored with a warning, and individual save files sync instead. Honouring it there would shuttle an empty card around while the saves the platform actually uses stopped syncing. + +PCSX2 only serves its card when Slot 1 holds a **Folder** card rather than a **File** card. With a File card the broker refuses the transfer and the session will not start, so set the card type before turning the flag on (see the [PCSX2 memory card setup](https://github.com/LoneAngelFayt/pcsx2-romm-integration#memory-card-setup)). Dolphin pins its own folder card, so GameCube needs no extra setup. Because each session starts from your library, the first time RomM uses a container that already has a card on it, it stops and asks what to do: @@ -56,37 +90,95 @@ RomM records your answer per container, so it only asks once. !!! warning "Playing on the container directly" Syncing only happens around a RomM streaming session. If you play on the emulator container directly, outside RomM, those saves and cards stay on the container and are not pulled into your library. A later RomM session loads your library's copy over them, so make your progress through RomM to keep it. -## Setup +## Multi-disc games + +A multi-disc game boots its playlist, and on `dc`, `saturn`, `segacd`, `turbografx-cd` and `dos` the mounted disc can be changed without restarting the emulator. Only the game's own playlist entries are valid swap targets. + +Whichever disc was mounted when a state was written is the disc remounted when that state is loaded, so a save made on disc 3 resumes on disc 3. + +PS2 has no swap route, but PCSX2's own UI can change discs from inside the stream. -### Run the emulator containers +## Joining a session -Pick the broker repos for the platforms you want and follow each repo's `docker-compose.yml`. Each container exposes two things we need: +A session can be opened to a second player, and any user can see which sessions are joinable and ask to join one. Joining attaches to the running session rather than claiming a container, so it doesn't consume a second one. -- The **Selkies web UI** the browser loads the stream from (an HTTPS port). -- The **broker API** RomM sends launch, save, and volume commands to (default port `8000`). +## Administration -### Setup `config.yml` +Two admin-only capabilities sit on top of the per-user session model: -Add a `streaming` block with one entry per emulator container, full schema in [Configuration File → `streaming`](../reference/configuration-file.md#streaming). +- **Desktop sessions** open a container's desktop with no game running, which is how you configure an emulator inside the container that will run it (BIOS paths, controllers, per-emulator settings). A desktop claims the container under the same lock as a game, so it blocks players and a running game blocks it. +- **The container fleet** lists every configured container, what it is currently running and who claimed it, and can force-release any session. It is one row per container rather than per platform, because a container serving five platforms still holds one session. A container RomM cannot claim (a `host` with no scheme, no reachable broker) is listed as unconfigured rather than idle, so the misconfiguration is visible. -- `host` must be reachable from clients and served over **HTTPS** (Selkies WebRTC requires a secure context). Use the container's built-in self-signed cert or a [reverse proxy with TLS](../install/reverse-proxy.md). -- `broker_host` is called server-side, so HTTP is fine. If the containers share a Docker network, use the container name (e.g. `http://pcsx2:8000`). If `broker_host` is omitted, it gets derived from `host`. -- `label` is the text shown on the play action. -- `memory_card_sync: true` opts a **PS2** or **GameCube** container into whole-card sync (see [Memory cards](#memory-cards)). It has no effect on platforms without a memory card. -- `library_path` overrides the in-container library path if the container mounts the RomM library somewhere other than the default `/romm/library`. -- `emulator` sets an explicit name used to group this container's states and memory cards, lowercased. It defaults to `label`, then the platform slug, so setting it keeps stored states and cards attached when you rename a label later. +Force-releasing is also how you recover a platform stuck as in use when an owner disconnected without releasing it. -Multiple platforms can share one container (point `ngc` and `wii` at the same Dolphin instance) or each use their own. +## Setup + +### Run the webstation container + +Stand up [docker-webstation](https://github.com/linuxserver/docker-webstation) with [romm-broker](https://github.com/romm-streaming/romm-broker), mounting your ROM library read-only. The container exposes two things RomM needs: + +- The **Selkies web UI** the browser loads the stream from. +- The **broker API** RomM sends launch, save, state and disc commands to, served under the container's `SUBFOLDER` on the same origin. + +A worked compose file lives in [`docker-compose.streaming.yml`](https://github.com/rommapp/romm/blob/master/docker-compose.streaming.yml) upstream. The image is amd64-only. + +### Set up `config.yml` + +Add a `streaming` block with **one entry per container**, full schema in [Configuration File → `streaming`](../reference/configuration-file.md#streaming). Container-level keys are the defaults for every platform it serves, and a platform block only names what differs. + +```yaml +streaming: + enabled: true + containers: + - protocol: webstation + host: https://192.168.1.56:3010 + subfolder: /streaming + library_path: /romm + broker_secret: change-me + label: Emulation station + platforms: + snes: retroarch # just the emulator name... + ps2: # ...or a block overriding container keys + emulator: pcsx2 + label: PCSX2 + memory_card_sync: true + ngc: + emulator: dolphin + label: Dolphin + memory_card_sync: true +``` + +- `host` is browser-facing and must be served over **HTTPS**, since Selkies WebRTC needs a secure context. Use the container's self-signed cert or a [reverse proxy with TLS](../install/reverse-proxy.md). A path (`/streaming`) works too when the container is reverse proxied onto RomM's own origin, in which case you have to name `broker_host` yourself, because a bare path carries no address RomM can dial. +- `subfolder` matches the container's `SUBFOLDER` and is what the broker routes hang off. +- `broker_host` is called server-side, so HTTP is fine. Omitted, a webstation container derives it from `host`. Pool members are identified by their broker host, so two containers pooling for a platform need distinct ones. +- `library_path` is where the container sees your ROM library, when that differs from RomM's own `/romm/library`. Keep it stable across a container rebuild so state and save history stays attached. +- `emulator` names the state and memory-card namespace. Set it explicitly, so renaming a `label` later doesn't orphan stored states and cards. ### Set the shared secret -`STREAMING_BROKER_SECRET` authenticates calls to the broker. Set the **same value** in every container. If a broker needs a different secret, set `broker_secret` on that entry in `config.yml` and leave `STREAMING_BROKER_SECRET` unset, because the env var wins over the per-container value whenever it carries one. +`STREAMING_BROKER_SECRET` authenticates calls to the broker, and has to match the container's `BROKER_SECRET`. Set the **same value** on every container. If a broker needs a different secret, set `broker_secret` on that entry in `config.yml` and leave `STREAMING_BROKER_SECRET` unset, because the env var wins over the per-container value whenever it carries one. + +### Tune the timeouts -If a broker's save wait exceeds the default 45 seconds, raise `STREAMING_SAVE_TIMEOUT` (seconds) so Save & Exit doesn't time out. `STREAMING_STATE_HISTORY_LIMIT` (default `50`) caps how many save states RomM keeps per game, emulator, and user before pruning the oldest, and `0` keeps every state. All are set as env vars (see [Environment Variables](../reference/environment-variables.md)). +All set as env vars (see [Environment Variables](../reference/environment-variables.md)): + +| Variable | Default | What it bounds | +| ------------------------------- | ------- | ------------------------------------------------------------------------------------- | +| `STREAMING_LAUNCH_TIMEOUT` | `600` | How long a launch may take, covering package and archive extraction before boot | +| `STREAMING_SAVE_TIMEOUT` | `45` | How long a save-and-exit may take, raise it if a broker's `SAVE_WAIT` exceeds it | +| `STREAMING_STATE_HISTORY_LIMIT` | `50` | Save states kept per ROM, emulator and user before the oldest are pruned, `0` for all | ## Troubleshooting -- **No Play on `