[Bug] fork() called from a process already using GNU OpenMP, this is unsafe.

[quentinblampey]

I got this issue when using fast_array_utils>=1.4.1 and torch DataLoaders with multiple workers. It seems the new fast_array_utils njit decorator is conflicting with torch.

EDIT: see below an MRE.

[quentinblampey]

Minimum reproducible example

I took some time to investigate more, and I came with a minimum reproducible example involving only scanpy/torch/fast_array_utils:

import scanpy as sc
import torch
from anndata import AnnData
from fast_array_utils.conv import to_dense
from torch import Tensor
from torch.utils.data import DataLoader, Dataset


class DummyDataset(Dataset):
    def __init__(self, adata: AnnData) -> None:
        super().__init__()
        self.adata = adata

    def __len__(self) -> int:
        return self.adata.n_obs

    def __getitem__(self, index: int) -> Tensor:
        return torch.tensor(to_dense(self.adata.X[:1], to_cpu_memory=True))  # notice the usage of `to_dense` here


if __name__ == "__main__":
    adata = sc.datasets.pbmc3k()
    sc.pp.normalize_total(adata)  # weirdly, this is needed to reproduce the issue

    dataset = DummyDataset(adata=adata)

    loader = DataLoader(
        dataset,
        batch_size=256,
        shuffle=False,
        drop_last=False,
        num_workers=4,  # >0 workers is important to repoduce the issue
    )

    # this is the line causing the crash (it triggers the creation of batches with workers in parallel)
    for batch in loader:
        break

It crashes with:

Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
RuntimeError: DataLoader worker (pid(s) 1494, 1495, 1496, 1497) exited unexpectedly

OS and versions

My setup is a bit weird: I'm using Coder, hosted by Kubernetes (I can double-check the details with my team if it matters).
It's on Ubuntu 24.04.4 LTS, and I'm using fast_array_utils==1.4.1, anndata==0.12.17, numba==0.65.1, and torch==2.12.1.

Numba threading information:

__Threading Layer Information__
TBB Threading Layer Available                 : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available              : True
+-->Vendor: GNU
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

[flying-sheep]

could you please add a traceback via faulthandler.enable()?

[quentinblampey]

Should I just add this at the beginning of the script? I did that, but it seems not to change the full traceback, see below.

Full traceback

Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Traceback (most recent call last):
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1287, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/shared/coder/quentin.blampey/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/multiprocessing/queues.py", line 111, in get
    if not self._poll(timeout):
           ~~~~~~~~~~^^^^^^^^^
  File "/shared/coder/quentin.blampey/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File "/shared/coder/quentin.blampey/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 440, in _poll
    r = wait([self], timeout)
  File "/shared/coder/quentin.blampey/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 1148, in wait
    ready = selector.select(timeout)
  File "/shared/coder/quentin.blampey/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/selectors.py", line 398, in select
    fd_event_list = self._selector.poll(timeout)
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
    _error_if_any_worker_fails()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: DataLoader worker (pid 8574) is killed by signal: Terminated. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/shared/coder/quentin.blampey/dev/novae/repro.py", line 37, in <module>
    for batch in loader:
                 ^^^^^^
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 718, in __next__
    data = self._next_data()
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1501, in _next_data
    idx, data = self._get_data()
                ~~~~~~~~~~~~~~^^
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1460, in _get_data
    success, data = self._try_get_data()
                    ~~~~~~~~~~~~~~~~~~^^
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1300, in _try_get_data
    raise RuntimeError(
        f"DataLoader worker (pid(s) {pids_str}) exited unexpectedly"
    ) from e
RuntimeError: DataLoader worker (pid(s) 8574, 8575, 8576, 8577) exited unexpectedly

[ilan-gold]

@quentinblampey What would you expect here as a solution? We ran into this in https://annbatch.readthedocs.io/en/stable/ and basically just went with "don't do that" (i.e., use spawn or better yet, don't use DataLoader at all since annbatch is threaded). I'm not sure how we can prevent people from forking process that have thread resources.

[quentinblampey]

Hello @ilan-gold,

I mostly use fast-array-utils because it's agnostic to the array type, which is very convenient for being very flexible with user preferences. Yet, I don't really use its parallelization (actually, I didn't know it was parallelized until I faced this issue).

So, one solution could be to have some settings to disable parallelization - do you think this would be possible?
For some context: most users use this package (novae) in inference mode only, i.e., we can use it for each spatial slide sequentially, so we load a maximum of 1 million cells before moving on to the next slide. This easily fits in memory with sparse adata.X and allows handling large datasets of 50M+ cells.

NB: the above strategy obviously doesn't work for model training, but at least it's not a user-related use case.

Another potential solution: I could take a deeper look into annbatch and see if I can also get rid of the DataLoader. It may be complex though, because I use DataLoader of graphs (pytorch geometric), so I'm not sure if I can directly apply annbatch for this.

For now, I pinned fast-array-utils<1.4.0 until I find a solution.

[flying-sheep]

one solution could be to have some settings to disable parallelization

we’re already laboriously doing that in certain situations:

fast-array-utils/src/fast_array_utils/numba/__init__.py

Lines 115 to 124 in c53c8ea

	if _is_in_unsafe_thread_pool(): # pragma: no cover
	msg = f"Detected unsupported threading environment. Trying to run {f.__name__} in serial mode. In case of problems, install `tbb`."
	elif _needs_parallel_runtime_probe() and not _parallel_numba_runtime_is_safe():
	msg = (
	f"Detected an unsupported numba parallel runtime. Running {f.__name__} in serial mode as a workaround. "
	"Set `NUMBA_THREADING_LAYER=workqueue` or install `tbb` to avoid this fallback."
	)
	if not (run_parallel := msg is None):
	warnings.warn(msg, UserWarning, stacklevel=2)
	return fns[run_parallel](*args, **kwargs)

turning it off e.g. by setting an env var before importing it or so would certainly be a possible feature.

if there’s a reliable way to detect the situation you’re running into, we could add it to the pile here.

PS: I think the “In case of problems, install tbb” is misleading now that this works reliably. It should probably read “if you want to use parallelization, install tbb”.