Skip to content

Make job completion order deterministic, improve tests - #42

Merged
cottsay merged 2 commits into
masterfrom
cottsay/parallel-executor-fixes
Sep 11, 2026
Merged

cottsay merged 2 commits into
masterfrom
cottsay/parallel-executor-fixes

Conversation

@cottsay

@cottsay cottsay commented Aug 29, 2026

Copy link
Copy Markdown
Member

When multiple jobs complete in the same "tick" of the parallel execution loop, the done_futures set can result in different colcon return codes depending on which one happens to get processed first. A more desirable behavior is to prefer the return code of the job which was started first.

While it is currently unlikely to see multiple jobs finish in the same tick due to strict concurrency pre-limiting, this becomes much more common under dynamic scheduling and resource-guarded execution.

This change also includes an additional test for the parallel executor, and switches all of the job identifiers to use consistent identifiers, where they were previously using different values in the overall job dictionary from the identifier used on the job objects themselves.

Assisted-by: Gemini 3.5 Flash

When multiple jobs complete in the same "tick" of the parallel execution
loop, the `done_futures` set can result in different colcon return
codes depending on which one happens to get processed first. A more
desirable behavior is to prefer the return code of the job which was
started first.

While it is currently unlikely to see multiple jobs finish in the same
tick due to strict concurrency pre-limiting, this becomes much more
common under dynamic scheduling and resource-guarded execution.

This change also includes an additional test for the parallel executor,
and switches all of the job identifiers to use consistent identifiers,
where they were previously using different values in the overall job
dictionary from the identifier used on the job objects themselves.

Assisted-by: Gemini 3.5 Flash <gemini@google.com>
@cottsay cottsay self-assigned this Aug 29, 2026
@cottsay cottsay added the bug Something isn't working label Aug 29, 2026
@codecov-commenter

codecov-commenter commented Aug 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.38%. Comparing base (94dfeff) to head (7b5e008).

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #42      +/-   ##
==========================================
+ Coverage   96.73%   97.38%   +0.65%     
==========================================
  Files           4        4              
  Lines         153      153              
  Branches       41       41              
==========================================
+ Hits          148      149       +1     
  Misses          1        1              
+ Partials        4        3       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@knmcguire

Copy link
Copy Markdown
Contributor

I'll review this so add me as reviewer @knmcguire

@cottsay
cottsay requested a review from knmcguire August 31, 2026 16:23

@knmcguire knmcguire left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also have a test on the first addition you made?

Claude Opus 5.1 generated this:

class ImmediateJob(Job):

    def __init__(self, identifier, rc):
        super().__init__(
            identifier=identifier, dependencies=set(), task=None,
            task_context=None)
        self.rc = rc

    async def __call__(self, *args, **kwargs):
        return self.rc


def test_simultaneous_completion_prefers_first_started():
    extension = ParallelExecutorExtension()
    args = SimpleNamespace(parallel_workers=2)

    # repeat: set iteration order varies run to run, so a single
    # pass would only catch the bug ~50% of the time
    for _ in range(50):
        jobs = OrderedDict()
        jobs['early'] = ImmediateJob('early', 3)
        jobs['late'] = ImmediateJob('late', 7)
        rc = extension.execute(args, jobs, on_error=OnError.continue_)
        assert rc == 3, f'expected first-started rc 3, got {rc}'

I've tested it out on the current master and the patch, and indeed it passes on your fix.

FYI, I had to use Claude Opus 5.1 to check out a couple of assumptions for the review as here but I tested off of the changes myself.

Comment thread colcon_parallel_executor/executor/parallel.py
Comment thread test/test_executor_parallel.py
Assisted-by: Claude Opus 5.1 <noreply@anthropic.com>
@cottsay

cottsay commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Should we also have a test on the first addition you made?

Sure! Added verbatim in 7b5e008.

@cottsay
cottsay requested a review from knmcguire September 10, 2026 22:43

@knmcguire knmcguire left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! thanks!

@cottsay
cottsay merged commit bb4e553 into master Sep 11, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants