Prevent threaded Dask workers in DistRDF backend

[JAGANNATHANJP]

This PR prevents unsupported threaded Dask workers in DistRDF.

Distributed RDataFrame with Dask threads may lead to crashes and does not provide advantages due to Python GIL limitations. This change validates worker configuration at backend initialization and raises a RuntimeError when threaded workers are detected.

Suggested configuration:

• processes=True
• threads_per_worker=1

[github-actions]

Test Results

    23 files      23 suites   3d 13h 59m 54s ⏱️
 3 878 tests  3 878 ✅ 0 💤 0 ❌
79 003 runs  79 003 ✅ 0 💤 0 ❌

Results for commit 30314f0.

♻️ This comment has been updated with latest results.

[vepadulano]

Thank you @JAGANNATHANJP ! Before proceeding with the CI, I would like to request you to add a test for this new behaviour, perhaps in roottest/python/distrdf/backends/check_backend.py

[JAGANNATHANJP]

Added a test covering the case where scheduler_info() does not provide worker information during Dask backend initialization.

[JAGANNATHANJP]

Resolved the merge conflict with the latest master and updated the branch.

[vepadulano]

Hi @JAGANNATHANJP ,

Apparently there are still some conflicts, see e.g. https://github.com/root-project/root/actions/runs/28595013336/job/85566036258?pr=22393

Auto-merging roottest/python/distrdf/backends/check_backend.py
  CONFLICT (content): Merge conflict in roottest/python/distrdf/backends/check_backend.py
  error: could not apply def6df24ab... Add tests for Dask worker validation

[vepadulano]

Dear @JAGANNATHANJP ,

The test suite is failing on all platforms with the following error

 >       with pytest.warns(
              UserWarning, match="Number of partitions 2 is greater than number of entries 1 in the dataframe"
          ):
  E       Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
  E        Emitted warnings: [].
  
  ../../../../../src/roottest/python/distrdf/backends/check_backend.py:161: Failed
  =========================== short test summary info ============================
  FAILED ../../../../../src/roottest/python/distrdf/backends/test_all.py::TestInitialization::test_initialization_method[dask] - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
   Emitted warnings: [].
  !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
  ========================= 1 failed, 5 passed in 4.32s ==========================

Did you run the test suite locally before resubmitting your changes?

[JAGANNATHANJP]

Hi @vepadulano,

Apologies for that! The failure was caused by an accidental indentation error during the last merge conflict resolution (df.GetValue() got shifted outside the pytest.warns block).

I have fixed the indentation in commit 9a58aa7 and verified the full test suite locally inside my build environment. All 14 tests in check_backend.py (including the new Dask worker validation test and test_initialization_method) passed 100% cleanly locally.

Could you please approve running the CI checks when you have a moment? Thank you!

[JAGANNATHANJP]

Hi @vepadulano,

I investigated the recent CI failures. The failure on fedora44 (pyunittests-roottest-python-distrdf-backends-all) was caused by the Dask LocalCluster attempting to spin up a web dashboard on port 8787 during the test, which crashed on the CI runner due to port conflicts/restrictions.

I have pushed a commit to explicitly pass dashboard_address=None in check_backend.py to ensure it runs completely silently.

(Note: The other failures on Alma8/Alma9/Mac appear to be unrelated upstream C++ IO and CMake flakes).

Could you please review when the CI finishes? Thanks!

[vepadulano]

Thanks! There are still some things to cleanup.

[JAGANNATHANJP]

Hi @vepadulano,

I have addressed all your review comments and also fixed the 3 CI failures from the previous run. The failures were caused by a pytest.raises(...) line in check_backend.py exceeding the ruff line length limit, which is now fixed in commit c446a5b.

Could you please review when the CI finishes? Thank you!

[vepadulano]

Thanks for this nice piece of work @JAGANNATHANJP ! I went through the changes and I believe they look good. The last remaining test failure was just a warning raised by the dask cluster finding the same address used twice, which I fixed using the :0 syntax in the LocalCluster initialization to use a random address to avoid tripping Dask. I am running the CI now to make sure the tests pass on all platforms. No further action is required from your side, thanks again!

[JAGANNATHANJP]

Thank you so much @vepadulano for the review, guidance, and help with the cluster address fix! Glad to contribute to ROOT!