Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions design/native.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Native process spawning (`use-native` flag)

The `use-native` flag uses a process-spawning implementation that does not
depend on the `process` package. It crashes intermittently. This note
records an investigation (2026-10-18) of the cause and of whether the
approach can work.

## Can it work?

Processes can be spawned without the `process` package. They cannot be
spawned the way the branch does it, by running Haskell code in a
`forkProcess` child under the threaded RTS. The tests are built with
`-threaded -with-rtsopts=-N`. After `fork()`, the child keeps only the
thread that called fork. All other Haskell threads and RTS workers are
gone, but any lock they held stays locked. The `unix` docs warn about this
for `forkProcess`.

The `process` package avoids this by doing fork/exec in C
(`cbits/runProcess.c`, using `vfork` or `posix_spawn`). It never runs
Haskell code between fork and exec, and it creates its pipes close-on-exec
while holding a lock.

## Likely causes

Two small programs using only the `unix` package, `Repro.hs` and
`Leak.hs`, modelled parts of `Posix.hs`. The programs were not kept.

1. **The debug logging deadlocks the child.** `hPutStrLn stderr` in
`childAction` and `exec` takes the `stderr` Handle's MVar. If another
thread held that MVar at fork time, the child blocks forever.
- With 4 threads × 200 spawns and other threads writing to `stderr`, a
child that writes to `stderr` hung in 20 of 20 runs.
- Without the writes, 20 of 20 runs passed.
- The debug code therefore causes hangs of its own, which can hide the
original crash. The `/proc/<pid>/fd` listing in the child
(`Dir.readFiles`) has the same problem: it allocates, can trigger a
GC, and uses Handles.
2. **Pipe fds leak into other children.** `createPipe` does not set
`O_CLOEXEC`, and nothing serialises spawns. When two spawns overlap,
child A inherits the pipe fds created for child B and keeps them after
`exec`. `Leak.hs` showed a pipe created by another thread (fds 17 and
18) still open in the exec'd `/bin/sh`. While A is alive:
- B never sees EOF on its stdin.
- The parent never sees EOF on B's stdout.

This only happens when spawns overlap, so the failures are
intermittent.
3. **Any other Haskell code in the child can deadlock the same way as
cause 1.** Examples:
- an exception in `action`;
- `executeFile` failing (the `XXX` comment in `newProcess`);
- a GC in the child.

If the child's Haskell code returns instead of exec'ing, the child also
runs RTS shutdown, which may flush stdio buffers copied from the parent
and duplicate output. This was not tested.

The actual crash was not reproduced. Whether it is a segfault, a hang,
`BlockedIndefinitelyOnMVar` or a wrong exit code would indicate which of
these causes it is.

## Recommended approach

Spawn in C, or call `posix_spawn(3)` through the FFI:

- `posix_spawn_file_actions_adddup2` / `addclose` replace `childAction`.
- `POSIX_SPAWN_SETSIGMASK` and `POSIX_SPAWN_SETSIGDEF` reset the signal
mask and the signal dispositions.
- An exec failure is returned to the parent as an errno. This replaces the
planned exception pipe.
- Create pipes with `pipe2(O_CLOEXEC)`. `dup2` onto fds 0, 1 and 2 clears
close-on-exec on the target fd, so the child still gets its stdio.

glibc implements `posix_spawn` with `clone(CLONE_VM|CLONE_VFORK)`, which is
safe in a multithreaded process. Wrapping the spawn in `runInBoundThread`
(`createProc''` in the debug commit) does not help, because the other
threads still exist at fork time.

## Build status at the time of the investigation

- `use-native` was rebased onto master as one commit, `Add debug logging`.
- `flake.nix` points `streamly-packages` at revision `69728978…`, the
revision used by streamly's flake. This provides prebuilt
streamly-0.11.0 and streamly-core-0.3.0.
- The `-f use-native` build fails:
- The debug code does not compile against streamly-core 0.3, because the
`Dir.readFiles` API changed.
- Without the debug code, the native path on master does not compile
either. `standalone` uses `withCreateProcess` and the `std_in` field,
but the native `Config` is only a `Bool`.

Because of these build failures, the test suite was not run with
`use-native`. The findings above come from the two small programs, not
from streamly-process itself.
10 changes: 9 additions & 1 deletion src/Streamly/Internal/System/Process.hs
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ module Streamly.Internal.System.Process
)
where

import Control.Concurrent (forkIO)
import Control.Concurrent (forkIO, runInBoundThread)
import Control.Exception (Exception(..), catch, throwIO)
import Control.Monad (void, unless)
import Control.Monad.Catch (MonadCatch, throwM)
Expand Down Expand Up @@ -616,6 +616,14 @@ createProc' modCfg path args = do

Config cfg = modCfg $ mkConfig path args

createProc'' ::
(Config -> Config) -- ^ Process attribute modifier
-> FilePath -- ^ Executable path
-> [String] -- ^ Arguments
-> IO (Maybe Handle, Maybe Handle, Maybe Handle, ProcessHandle)
createProc'' modCfg path args =
runInBoundThread $ createProc' modCfg path args

{-# INLINE putChunksClose #-}
putChunksClose :: MonadIO m =>
Handle -> Stream m (Array Word8) -> Stream m a
Expand Down
50 changes: 38 additions & 12 deletions src/Streamly/Internal/System/Process/Posix.hs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{-# LANGUAGE Safe #-}
-- {-# LANGUAGE Safe #-}
-- |
-- Module : Streamly.Internal.System.Process.Posix
-- Copyright : (c) 2020 Composewell Technologies
Expand Down Expand Up @@ -35,16 +35,19 @@ import Data.Tuple (swap)
import GHC.IO.Device (IODeviceType(..))
import GHC.IO.Encoding (getLocaleEncoding)
import GHC.IO.Handle.FD (mkHandleFromFD)
import System.IO (IOMode(..), Handle)
import System.IO (IOMode(..), Handle, hPutStrLn, stderr)
import System.IO.Error (isDoesNotExistError)
import System.Posix.IO (createPipe, dupTo, closeFd)
import System.Posix.Process (forkProcess, executeFile, ProcessStatus)
import System.Posix.Process (forkProcess, executeFile, ProcessStatus, getProcessID)
import System.Posix.Types (ProcessID, Fd(..), CDev, CIno)
import System.Posix.Signals (signalProcess, sigTERM)
import System.Posix.Internals (fdGetMode)
import qualified Streamly.Internal.FileSystem.Dir as Dir
import qualified Streamly.Prelude as Stream

import qualified GHC.IO.FD as FD
import qualified System.Posix.Process as Posix
import Data.List (intercalate)

-------------------------------------------------------------------------------
-- Utilities to create stdio handles
Expand All @@ -54,13 +57,13 @@ import qualified System.Posix.Process as Posix
-- We have to put the FDs into binary mode on Windows to avoid the newline
-- translation that the CRT IO library does.
setBinaryMode :: FD.FD -> IO ()
#if defined(mingw32_HOST_OS)
setBinaryMode fd = do
_ <- setmode (FD.fdFD fd) True
return ()
#else





setBinaryMode _ = return ()
#endif


-- See Posix.fdToHandle and GHC.IO.Handle.FD.fdToHandle
-- See stdin, stdout, stderr in module GHC.IO.Handle.FD
Expand Down Expand Up @@ -112,9 +115,18 @@ mkPipeDupChild :: Direction -> Fd -> IO (Fd, (IO (), IO (), IO ()))
mkPipeDupChild direction childFd = do
let setDirection = if direction == ParentToChild then id else swap
(child, parent) <- fmap setDirection createPipe
pid <- getProcessID
let parentAction = closeFd child
childAction =
closeFd parent >> void (dupTo child childFd) >> closeFd child
hPutStrLn stderr ("closing parent fd" ++ show parent) >>
closeFd parent >>
hPutStrLn stderr ("closed parent fd" ++ show parent) >>
hPutStrLn stderr ("duplicating child to fd" ++ show (child, childFd)) >>
void (dupTo child childFd) >>
hPutStrLn stderr ("duplicated child to fd" ++ show (child, childFd)) >>
hPutStrLn stderr ("closing child" ++ show child) >>
closeFd child >>
hPutStrLn stderr ("closed child" ++ show child)
failureAction = closeFd child >> closeFd parent
return (parent, (parentAction, childAction, failureAction))

Expand Down Expand Up @@ -144,7 +156,15 @@ mkStdioPipes pipeStdErr = do
-}

let parentAction = inpParent >> outParent >> errParent -- >> excParent
childAction = inpChild >> outChild >> errChild -- >> excChild
childAction =
hPutStrLn stderr "child input action doing"
>> inpChild
>> hPutStrLn stderr "child input action done"
>> hPutStrLn stderr "child output action doing"
>> outChild
>> hPutStrLn stderr "child output action done"
>> errChild -- >> excChild
-- childAction = inpChild >> outChild >> errChild -- >> excChild
failureAction = inpFail >> outFail >> errFail -- >> excFail

inpH <- toHandle WriteMode inp
Expand Down Expand Up @@ -238,6 +258,7 @@ newProcess ::
-> IO Process
newProcess action path args env = do
pid <- forkProcess exec
hPutStrLn stderr ("parent process " ++ show pid)
pidToProcess pid Nothing

where
Expand All @@ -246,7 +267,12 @@ newProcess action path args env = do
-- to the parent and clean up the parent fds. We can send the exceptions
-- via a pipe like we do for threads.
--
exec = action >> executeFile path True args env
exec = do
pid <- getProcessID
hPutStrLn stderr ("child process " ++ show pid)
fds <- Stream.toList . Stream.unfold Dir.readFiles $ ("/proc/" ++ show pid ++ "/fd")
hPutStrLn stderr (intercalate ", " fds)
action >> executeFile path True args env

newtype ProcessDoesNotExist = ProcessDoesNotExist ProcessID deriving Show

Expand Down
Loading