[RF] Make multiprocess gradient parallelization work with new vectorizing CPU backend - #23321
Open
guitargeek wants to merge 4 commits into
Open
[RF] Make multiprocess gradient parallelization work with new vectorizing CPU backend#23321guitargeek wants to merge 4 commits into
guitargeek wants to merge 4 commits into
Conversation
guitargeek
force-pushed
the
parallel-gradient
branch
from
September 9, 2026 14:36
0aaf736 to
ae8afbc
Compare
The evaluator-backed RooUnbinnedL computed the negative log-likelihood from the batch of probabilities with a scalar std::log loop, costing several times more than one vectorized likelihood pass now that the rest of the evaluation is SIMD. Use the same RooBatchCompute::reduceNLL() reduction as RooNLLVarNew, which also reproduces the RooNaNPacker-based error propagation of the loop (badness packed into the returned NaN). 🤖 Done with the help of AI
The ModularL branch of createNLL never passed the parsed EvalBackend on to the NLLFactory, whose default is the legacy backend. As a result, every parallel fit (fitTo with Parallelize(), which implies ModularL) silently evaluated the deprecated legacy scalar likelihood on the workers, costing about 7x per likelihood evaluation compared to the vectorized CPU backend and making the parallel gradient lose against any serial fit (benchmark: 8-channel unbinned simultaneous fit with 64 correlated constrained systematics, 96 free parameters, 200k events: serial fitTo 9.9s, Parallelize(8) 23.7s before, 4.7s after this and the accompanying gradient-job commits). The bitwise legacy-vs-modular comparisons in testLikelihoodGradientJob now request EvalBackend(Legacy) explicitly on the modular side: they compare against a legacy reference fit, so both likelihoods must use the same arithmetic (previously that happened by virtue of this bug). 🤖 Done with the help of AI
Every gradient calculation in LikelihoodGradientJob started with each worker evaluating the full likelihood once at the central point, only to obtain the scalar function value that NumericalDerivator needs for its step-size tolerances (SetupDifferentiate). Minuit already knows this exact value: the line search that precedes each gradient request stores it in MinimumParameters::Fval(). Hand that value through a new fifth argument of FCNBase::GradientWithPrevResult() (a backward-compatible overload that falls back to the old virtual), broadcast it to the workers along with the rest of the minimizer state, and pre-seed the derivator's existing central-value cache (fVxFValCache) with it, so SetupDifferentiate skips its function call. When the likelihood offsets changed in the same state update, NaN is broadcast instead and the workers evaluate as before, since the known value corresponds to the previous offsets. This is bitwise-transparent: the exact-equality comparisons in testLikelihoodGradientJob pass unchanged with the pre-seeding active. 🤖 Done with the help of AI
One gradient task used to be one partial derivative, so a gradient call in LikelihoodGradientJob costed one dequeue round trip through the queue process plus one result message per parameter. With the vectorized CPU backend a partial derivative is only a few milliseconds of work, and for fits with around a hundred parameters this scheduling overhead dominated the parallel gradient and capped its speedup. Each task now covers a strided block of parameters (indices congruent to the task id), so parameters with expensive derivatives - typically adjacent blocks of correlated systematics - spread evenly over the tasks, and the per-task results travel in one message. The number of tasks is configurable through the new Config::LikelihoodGradientJob::defaultNParamTasks; the automatic default of four tasks per worker keeps enough granularity for the queue to balance load between workers. Benchmark (96-parameter unbinned simultaneous fit, 8 channels x 25k events, medians of 3 interleaved repeats on 12 cores): Parallelize(8) 9.5s with per-parameter tasks, 4.7s with the automatic chunking, against 9.8s for the serial fit. 🤖 Done with the help of AI
guitargeek
force-pushed
the
parallel-gradient
branch
from
September 9, 2026 21:36
ae8afbc to
1e28d69
Compare
Test Results 23 files 23 suites 3d 21h 6m 15s ⏱️ For more details on these failures, see this check. Results for commit 1e28d69. ♻️ This comment has been updated with latest results. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The RooFit gradient parallelization was still using the RooFit legacy evaluation backend, which heavily penalized it compared to the new default vectorizing CPU backend. This had to be migrated to give the gradient parallelization a chance in proving it's worth.
And indeed, it works quite well! One problem is that the new CPU backend is so fast per gradient direction that the scheduling overhead for multiprocessing doesn't amortize if you schedule the derivatives for all individual parameters. Therefore, a new mechanism was added to chunk multiple parameters in one task.
A toy model with 96 parameters over 8 channels nicely illustrates the performance benefits of the gradient parallelization now:
fitToParallelize(8), unpatched masterParallelize(1), patchedParallelize(2), patchedParallelize(4), patchedParallelize(8), patched (phase 2, 1 param/task)Parallelize(8), patched + chunked tasksSo for this toy model, you get a nice 2x with four processes.
This PR therefore brings the gradient parallelization back into the discussion space when deciding how to make RooFit faster.