Skip to content

test: spread masternode-heavy functional tests apart in test_runner.py to fix asan job - #7658

Open
knst wants to merge 1 commit into
dashpay:developfrom
knst:fix-asan-reorder-tests
Open

test: spread masternode-heavy functional tests apart in test_runner.py to fix asan job#7658
knst wants to merge 1 commit into
dashpay:developfrom
knst:fix-asan-reorder-tests

Conversation

@knst

@knst knst commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Issue being fixed or feature implemented

ASan CI job was OOM-killed [guess, no proof] - https://github.com/dashpay/dash/actions/runs/33900161711/job/101121790181?pr=7646

As tested locally [with no asan, numbers are estimation because my workstation hardware configuration and CI are not the same anyway]:

The first biggest peak 4426 MB PSS / 6123 MB RSS with 37 dashd at 345 s, all 37 belonging to the four named tests with feature_protx_version (15 dashd), feature_llmq_connections (11 dashd), feature_llmq_is_retroactive and feature_asset_locks

The second biggest peak is 3935 MB, 34 dashd: during feature_dip3_deterministicmns (13 dashd), feature_llmq_rotation.py (9 dashd), 2 instances of feature_llmq_simplepose (6+6).

What was done?

Moving some masternode functional tests within the list reduces the number of dashd processes that run on the same time with -j4.

How Has This Been Tested?

By simulation [done by Fable 5.1], expected memory peak should be significantly reduced: -j4 3900 → 3124 MB.
There's no regression for -j6 5327 → 5300 and -j8 7349 → 7003 [by simulation]

On CI for PR asan job succeed while on develop fails: https://github.com/dashpay/dash/actions/runs/34049601389/job/101534812733

Breaking Changes

N/A

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation
  • I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

ASan CI job was OOM-killed [guess, no proof] - https://github.com/dashpay/dash/actions/runs/33900161711/job/101121790181?pr=7646

As tested locally [with no asan]:

The first biggest peak 4426 MB PSS / 6123 MB RSS with 37 dashd at 345 s, all 37 belonging to the four named tests with feature_protx_version (15 dashd), feature_llmq_connections (11 dashd), feature_llmq_is_retroactive and feature_asset_locks

The second biggest peak is 3935 MB, 34 dashd: during feature_dip3_deterministicmns (13 dashd), feature_llmq_rotation.py (9 dashd), feature_llmq_simplepose (6+6).

-----
Moving some masternode functional tests within the list reduces the number of dashd processes that run on the same time with -j4.
By simulation [done by Fable 5.1], expected memory peak should be significantly reduced: -j4 3900 → 3124 MB.
There's no regression for -j6 5327 → 5300 and -j8 7349 → 7003 [by simulation]
@knst knst added this to the 24 milestone Sep 6, 2026
@knst
knst force-pushed the fix-asan-reorder-tests branch from fcaf385 to 7913c0e Compare September 6, 2026 19:12
@thepastaclaw

thepastaclaw commented Sep 6, 2026

Copy link
Copy Markdown

✅ Final review complete — no blockers (commit 7913c0e)
Last checked: 2026-09-07 15:50 UTC

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

Potential PR merge conflicts

This is advisory only. It does not block CI, but it marks PRs that will likely need a rebase depending on merge order.

If this PR merges first

These open PRs will likely need a rebase:

If these PRs merge first

This PR will likely need a rebase:

@coderabbitai

coderabbitai Bot commented Sep 6, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: 2e03fda2-c807-403d-84a1-07e08ae0c2a0

📥 Commits

Reviewing files that changed from the base of the PR and between 2d55eca and 7913c0e.

📒 Files selected for processing (1)
  • test/functional/test_runner.py

Included review availability: Your plan provides up to 4 included reviews per hour; 2 remain after this review.


Walkthrough

The functional test runner reorders entries in the BASE_SCRIPTS list. Several LLMQ-related scripts move to earlier positions, and other existing entries change relative order. The set of test scripts remains unchanged.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 7913c

This updates only the default functional-test execution order to reduce concurrent masternode-heavy workloads. Test coverage remains unchanged, with no current merge-blocking risk identified.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the test ordering change, its purpose, and the affected file. It accurately reflects the main change.
Description check ✅ Passed The description explains the suspected ASan out-of-memory issue, the test reordering, the expected memory impact, and the testing results. It is directly related to the changeset.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fcaf38568b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread test/functional/feature_protx_version.py Outdated

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — GLM Flash + Sol

At exact head 7913c0e, the PR only reorders selected BASE_SCRIPTS entries, preserves the complete 316-entry multiset without duplicates or missing files, and works with the runner's front-to-back scheduling. Python syntax compilation passed, and no actionable correctness, coverage, or scheduling issues were found.

Source: reviewer 1: glm-5.3-flash (agent: phase1-reviewer, role: general); reviewer 2: glm-5.3-flash (agent: phase1-reviewer, role: dash-core-commit-history); reviewer 3: gpt-5.6-sol (agent: phase2-reviewer, role: general); reviewer 4: gpt-5.6-sol (agent: phase2-reviewer, role: dash-core-commit-history); final verifier: gpt-5.6-sol (agent: sol-verifier, role: final-verifier)

Review provenance

  • Phase 1 reviewers (GLM Flash): glm-5.3-flash — general (completed); agent phase1-reviewer, glm-5.3-flash — dash-core-commit-history (completed); agent phase1-reviewer
  • Fresh verifier (Sol): gpt-5.6-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers (Sol): gpt-5.6-sol — general (completed); agent phase2-reviewer, gpt-5.6-sol — dash-core-commit-history (completed); agent phase2-reviewer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants