Repository navigation
Let a job script size each worker's threads (0.9.0) - #14
Merged
Merged
Conversation
thread_budget() reads WORLDARRAY_THREADS ahead of OMP_NUM_THREADS and the job's slots, so a job script can give each dask worker its share of the job's cores where OMP_NUM_THREADS stays 1 for numpy and LSB_DJOB_NUMPROC counts the whole job. A handle opened on the driver on the shared context is sized in each worker by that worker's value. A value that is not a positive integer raises ValueError. shared_context(file_io_floor=) sets the least the file I/O pool takes under the budget (default FILE_IO_FLOOR, 8; 1 keeps file I/O at the budget). It is part of the context's key and its pickle recipe, and is ignored with threads=N or threads=0. No default changes. Claude-Session: https://claude.ai/code/session_0141wgF8mkGbjGfBFKfKhp64
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cellmap-analyze asked for this, for its dask workers. A worker's tensorstore pools should follow the CPUs the job gives it, not stay at 1 because
OMP_NUM_THREADSis 1 for numpy. The datasets are opened on the driver before the cluster starts, so the number has to be read in each worker.WORLDARRAY_THREADS.thread_budget()reads it ahead ofOMP_NUM_THREADSand the job's slots. A job script exports, for example,max(1, ncpus // processes). A handle onshared_context(...)pickles asthreads=None, so each worker rebuilds it with its own value. Anything but a positive integer raisesValueError, because no other program sets this variable. A blank value counts as unset. It's read when a context is first made, not at import.shared_context(file_io_floor=). The least the file I/O pool takes under the thread budget, 8 by default as in 0.8.1keeps file I/O equal to the budget, which is what cellmap-analyze wants until 8 has been measured on /nrs at scale. It's part of the context's key and its pickle recipe, and ignored withthreads=Northreads=0, so it never makes a second context there. 0.6 to 0.8 pickles still load.No default changes: with the variable unset and no
file_io_floor, every context is what 0.8 made.Tests: the variable's precedence and its bad values; the floor at 1, 0, 8 and 16, its key and recipe; and a handle pickled on the driver and sized in a spawned worker by that worker's
WORLDARRAY_THREADS. 192 pass on tensorstore 0.1.78 and on 0.1.68, the minimum. The lost-wakeup stress test passes, andmkdocs build --strictis clean.https://claude.ai/code/session_0141wgF8mkGbjGfBFKfKhp64