Skip to content

[ci] feat: implement a bot to suggest running tests before merging#14099

Closed
sayakpaul wants to merge 2 commits into
mainfrom
suggest-tests
Closed

[ci] feat: implement a bot to suggest running tests before merging#14099
sayakpaul wants to merge 2 commits into
mainfrom
suggest-tests

Conversation

@sayakpaul

Copy link
Copy Markdown
Member

Example comment:

Suggested slow tests to run before merge:

- tests/lora/
- tests/models/transformers/
- tests/pipelines/flux/

Where the diff in a PR maps cleanly to core modules and their corresponding tests (peft.py -> LoRA-related tests), we use a deterministic system for the suggestion. However, when that gets convoluted, we also use Qwen3.5-35B-A3 model through HF Inference Provider to rank and polish the suggestion.

@sayakpaul sayakpaul requested a review from yiyixuxu July 1, 2026 11:56
@github-actions github-actions Bot added CI size/L PR with diff > 200 LOC labels Jul 1, 2026
@yiyixuxu yiyixuxu requested a review from DN6 July 6, 2026 22:56
@yiyixuxu

yiyixuxu commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

@DN6 can you take a look at this PR?

down the road, we can streamline the workflow so final review will recommend a couple of tests in the serge UI, and the maintainer can just check a checkbox to run the slow test etc (not in the scope of this PR)

@DN6

DN6 commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

@sayakpaul Why not automatically run the recommended tests? Why require a manual trigger?

@sayakpaul

Copy link
Copy Markdown
Member Author

Recommended tests could be larger than intended. It would also interfere with the test fetcher mechanism (not just for pipelines but more generally) that we have been meaning to implement for a long time now. Additionally, in some cases, we might not even want to run the recommended slow/GPU TESTS. So, I wanted to keep it the scope very specific here.

@DN6

DN6 commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Also, we have a deterministic way to fetch tests in this workflow
https://github.com/huggingface/diffusers/blob/main/.github/workflows/pr_test_fetcher.yml

It fetches relevant tests to run based on which modules have been changed. We never activated it on PRs because we had some weird circular dependencies with SD1.5 in the library that would trigger the wrong tests to run. I think those have been fixed now.

I've been working updating the workflow here (it's almost ready to PR):
https://github.com/huggingface/diffusers/tree/update-test-fetcher

@sayakpaul

Copy link
Copy Markdown
Member Author

Yeah that is what I was referring to. Do you want to extend the workflow that would suggest maintainers to run tests that would not run in CI otherwise (outside fast tests for example)?

@sayakpaul

Copy link
Copy Markdown
Member Author

Discussed with @DN6 today and I am going to close this because we will have a robust test fetcher.

@sayakpaul sayakpaul closed this Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI size/L PR with diff > 200 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants