Build/Test Tools: Add plugin compatibility testing workflow.

[adamsilverstein]

Claude Code built this workflow, requirements and steering from me:

Adds a workflow that installs the top plugins from the WordPress.org directory one at a time against a given version of WordPress and checks that nothing fatals. The idea is to catch the WP Rocket class of breakage - a popular plugin fataling on every request against a new core version, taking real sites offline - while there is still time to fix core or reach the plugin author. See adamsilverstein#64 and https://x.com/austinginder/status/2090199834787541074

A Trac ticket is being opened for this and will be linked here once it exists. Core needs a ticket before anything can be committed, so this stays a draft until then.

How it works

plugin-compatibility.yml queries https://api.wordpress.org/plugins/info/1.2/ for the most popular plugins at run time, so there is no list to go stale. The API caps per_page at 250 and returns 250 without complaint for anything larger, so the fetch pages through the API and trims to the requested count, de-duplicating across pages because popularity ordering can shift between two requests. Counts above 1000 are rejected with a message rather than quietly truncated.

Shards are sized rather than counted: it aims for 25 plugins per shard up to 10 shards, so a run of 10 does not spin up five near empty jobs and a run of 250 is not squeezed into the same five. Past 250 the shards get longer instead of more numerous.

Each shard calls reusable-plugin-compatibility.yml, which installs WordPress with WP_DEBUG and WP_DEBUG_LOG on, starts php -S, and then for each plugin installs it, activates it, runs wp eval, requests / and /wp-login.php, checks wp-content/debug.log, and removes the plugin before moving to the next one.

WP_DEBUG_DISPLAY is left off on purpose so the site behaves the way a production site does - a fatal is an empty page and an HTTP 500 rather than a printed stack trace. The fatal error handler is disabled too, otherwise recovery mode swallows the fatal and deactivates the plugin mid-test.

Runs are workflow_dispatch (so a release lead can point it at a beta or RC as part of the pre-release checklist, and pick how many plugins to cover) and weekly against nightly. It is signal-only, not a check on every commit - a third party plugin breaking shouldn't turn core CI red on unrelated work.

What the first CI run turned up

The workflow ran on this PR through its own pull_request paths trigger, which was the point of including that trigger. Three of five shards passed and two failed, and the failures were worth having:

Four WooCommerce extensions (woocommerce-payments, google-listings-and-ads, woocommerce-paypal-payments, woocommerce-gateway-stripe) were reported as failed when core had simply refused to activate them - Requires Plugins: woocommerce, which is unmet by design when plugins are tested one at a time. Those are now recorded as SKIPPED. wp-reviews-plugin-for-google was reported as failed because it calls wp_safe_redirect() while loading, which makes WP-CLI exit non-zero without anything being broken. The wp eval check now only fails on an actual fatal.

instagram-feed was reported with the nonsense status HTTP 200000, which was a real bug - the curl fallback appended to output curl had already written. That is fixed by capturing the exit code separately. Digging into why the transfer stalled found something more interesting: WordPress spawned WP-Cron as a loopback request, and the single threaded built-in server could not answer it while still serving the request that spawned it, so the two deadlocked for the full 60 second curl timeout. WP-Cron is now disabled and the server gets PHP_CLI_SERVER_WORKERS.

Underneath that deadlock was a genuine fatal on nightly, which is exactly the class of thing this workflow is for:

Uncaught TypeError: preg_match(): Argument #2 ($subject) must be of type string, WP_Error given
  in wp-includes/functions.php:3089
#0 wp-includes/functions.php(3089): preg_match()
#1 wp-includes/media.php(4359): wp_check_filetype()
#2 .../instagram-feed/inc/class-sb-instagram-gdpr-integrations.php(159): wp_get_image_editor()

It fires from a cron callback, so with WP-Cron disabled the current checks no longer reach it. Running due cron events through WP-CLI would catch it deterministically and is worth doing as a follow up. It looks like it deserves its own ticket either way.

pull_request and push runs now test the latest stable release with 10 plugins instead of nightly with 100. Those runs exist to check that the workflow itself still works, and a real ecosystem fatal against nightly would otherwise sit as a red check on every later change to these two files. Scheduled and manually dispatched runs keep nightly and the full count, which is where the ecosystem signal belongs.

How has this been tested

actionlint 1.7.12 with shellcheck 0.10.0 on PATH reports 0 errors for both new files and 0 across the whole .github/workflows directory. The repo's own actionlint and Zizmor jobs pass. A separate zizmor code scanning check failed on a new unpinned-images alert for the database service image; since the image comes from db-type and db-version inputs it cannot be pinned to a digest, so it carries an inline # zizmor: ignore[unpinned-images] with a comment explaining why. zizmor 1.24.1 locally reports no findings after that.

The matrix builder was run locally against the live API. 10 gave 1 shard of 10, 100 gave 4 shards of 25, 250 gave 10 shards of 25, 300 gave 10 shards of 30 across 2 API pages, and 600 gave 10 shards of 60 across 3 API pages with 600 unique slugs. A count of 1001, abc, and 0 each exit 1 with a message.

The per-plugin loop was extracted from the YAML and run unchanged in a container with WP-CLI, PHP 8.3 and MySQL 8.4, which is as close to the runner as could be managed locally:

• The exact six plugins that failed or were misreported in the first CI run - three now SKIPPED for unmet dependencies, instagram-feed and wp-reviews-plugin-for-google now PASS, classic-editor still PASS, job exits 0.
• Four deliberately fataling test plugins, one per detection path, all still FAIL with the right reason: fatal on plugin load caught at activation, fatal on template_redirect caught as HTTP 500 on /, fatal on shutdown caught at activation, and a fatal in wp_footer (which still returns HTTP 200 because output already started) caught in debug.log. A healthy plugin in the same list still passed and cleanup left wp-content/plugins and active_plugins clean.
• A nonexistent slug is reported as SKIPPED and does not fail the job.
• The top 10 popular plugins against latest, matching what a pull_request run does - all 10 passed.

Not tested: multisite, PHP versions other than 8.3, MariaDB, and the Slack notification and failed-workflow jobs, which are copied from install-testing.yml unchanged.

Types of changes

• Add .github/workflows/plugin-compatibility.yml, the caller, which pages the WordPress.org API and fans out to a matrix sized to the plugin count.
• Add .github/workflows/reusable-plugin-compatibility.yml, which installs WordPress and tests one shard of plugins in isolation.
• Write a per-plugin results table to the workflow summary and fail the job only when a plugin fatals.
• Record an unmet Requires Plugins dependency, a WP-CLI redirect, and a failed download as skipped rather than failed.

Open questions

• Should this hook into .version-support-*.json for the PHP version instead of pinning 8.3, or is one current PHP version the right scope for a smoke test?
• Testing plugins one at a time means every WooCommerce extension is skipped, and those are a large slice of the popular list. Installing WooCommerce alongside them would cover more real sites but breaks the isolation that keeps one broken plugin from masking another. Worth a follow up?
• Premium plugins like WP Rocket itself can't be fetched from the .org API, so the exact incident that prompted this wouldn't have been caught. Is there an appetite for a vendor-supplied zip input down the road, or does that raise too many licensing questions?
• Does any of this overlap with what Plugin Check or Tide already do? Nothing there appears to run plugins against unreleased core, but confirmation from someone closer to that infrastructure would help.

AI Use

Code and description written with 🤖 Claude Code, working from acceptance criteria in the linked issue. I will review and test.

[jeffpaul]

Does any of this overlap with what Plugin Check or Tide already do? Nothing there appears to run plugins against unreleased core, but confirmation from someone closer to that infrastructure would help.

While Tide is still running, its been generally unsupported for years and I've long considered trying to get it shut down as I'm unsure how many entities are even making use of its available audit data set (with the exception of the PHP Compat Checker plugin). If there was interest in having this functionality more formally supported within Tide and then in consuming the resulting data, then that seems great to me but we'd likely want to get someone / a team / a sponsor to help ensure Tide continues to be supported after this implementation completes.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props adamsilverstein, jeffpaul, adrianduffell.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[adrianduffell]

Testing plugins one at a time means every WooCommerce extension is skipped, and those are a large slice of the popular list. Installing WooCommerce alongside them would cover more real sites but breaks the isolation that keeps one broken plugin from masking another. Worth a follow up?

This seems worth a follow-up to me given they make the popular list. So in general, if a plugin has Requires Plugins headers then pre-install the required plugins first to allow the original plugin to be tested.