Repository navigation
Add optional lazy plugin loading driven by entry points - #327
Merged
Merged
Conversation
PluginRegistry imported every plugin module at startup (~260 plugin modules, ~2.4s) even when a run uses one or two plugins. Add an opt-in lazy mode (--lazy-load, NODESCRAPER_LAZY_LOAD=1, or PluginRegistry(lazy_load=True)) that indexes plugin names from source without importing and resolves a plugin class on first access, i.e. when PluginExecutor runs it. Eager loading stays the default. - LazyPluginMapping: dict-like name->class mapping; broken plugins are logged and treated as missing instead of aborting the registry. - ConfigRegistry can defer entry-point recipes (lazy_entry_point_configs) so AllIbPlugins no longer resolves every plugin at startup. - In lazy mode the CLI builds per-plugin subparsers only for plugins named on the command line. - Centralize registry creation: cli.main() builds the single registry and installs it with set_default_registry(); PluginDiscovery and the PluginExecutor fallback use default_registry() instead of building their own. - Add scripts/memray to compare memory use of the eager and lazy paths. Co-Authored-By: Claude <noreply@anthropic.com>
github-actions
Bot
added
documentation
alexandraBara
approved these changes
Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in lazy plugin loading mode plugins no longer import every pluginmodule at startup. The default (eager) behaviour is unchanged.
Enable with
node-scraper --lazy-load ...orNODESCRAPER_LAZY_LOAD=1.Lazy plugin registry (
pluginregistry.py)PluginRegistry(lazy_load=...)option andLazyPluginMapping.mapping[name],get(name)andname in mappingimport just that plugin.CLI
--lazy-loadflag.run-pluginssubparsers are built only for the plugins named on the command line,and
--helpstill builds all of them sTest plan
NODESCRAPER_LAZY_LOAD=1and without it