diff --git a/design/native.md b/design/native.md new file mode 100644 index 00000000..5296880e --- /dev/null +++ b/design/native.md @@ -0,0 +1,94 @@ +# Native process spawning (`use-native` flag) + +The `use-native` flag uses a process-spawning implementation that does not +depend on the `process` package. It crashes intermittently. This note +records an investigation (2026-10-18) of the cause and of whether the +approach can work. + +## Can it work? + +Processes can be spawned without the `process` package. They cannot be +spawned the way the branch does it, by running Haskell code in a +`forkProcess` child under the threaded RTS. The tests are built with +`-threaded -with-rtsopts=-N`. After `fork()`, the child keeps only the +thread that called fork. All other Haskell threads and RTS workers are +gone, but any lock they held stays locked. The `unix` docs warn about this +for `forkProcess`. + +The `process` package avoids this by doing fork/exec in C +(`cbits/runProcess.c`, using `vfork` or `posix_spawn`). It never runs +Haskell code between fork and exec, and it creates its pipes close-on-exec +while holding a lock. + +## Likely causes + +Two small programs using only the `unix` package, `Repro.hs` and +`Leak.hs`, modelled parts of `Posix.hs`. The programs were not kept. + +1. **The debug logging deadlocks the child.** `hPutStrLn stderr` in + `childAction` and `exec` takes the `stderr` Handle's MVar. If another + thread held that MVar at fork time, the child blocks forever. + - With 4 threads × 200 spawns and other threads writing to `stderr`, a + child that writes to `stderr` hung in 20 of 20 runs. + - Without the writes, 20 of 20 runs passed. + - The debug code therefore causes hangs of its own, which can hide the + original crash. The `/proc//fd` listing in the child + (`Dir.readFiles`) has the same problem: it allocates, can trigger a + GC, and uses Handles. +2. **Pipe fds leak into other children.** `createPipe` does not set + `O_CLOEXEC`, and nothing serialises spawns. When two spawns overlap, + child A inherits the pipe fds created for child B and keeps them after + `exec`. `Leak.hs` showed a pipe created by another thread (fds 17 and + 18) still open in the exec'd `/bin/sh`. While A is alive: + - B never sees EOF on its stdin. + - The parent never sees EOF on B's stdout. + + This only happens when spawns overlap, so the failures are + intermittent. +3. **Any other Haskell code in the child can deadlock the same way as + cause 1.** Examples: + - an exception in `action`; + - `executeFile` failing (the `XXX` comment in `newProcess`); + - a GC in the child. + + If the child's Haskell code returns instead of exec'ing, the child also + runs RTS shutdown, which may flush stdio buffers copied from the parent + and duplicate output. This was not tested. + +The actual crash was not reproduced. Whether it is a segfault, a hang, +`BlockedIndefinitelyOnMVar` or a wrong exit code would indicate which of +these causes it is. + +## Recommended approach + +Spawn in C, or call `posix_spawn(3)` through the FFI: + +- `posix_spawn_file_actions_adddup2` / `addclose` replace `childAction`. +- `POSIX_SPAWN_SETSIGMASK` and `POSIX_SPAWN_SETSIGDEF` reset the signal + mask and the signal dispositions. +- An exec failure is returned to the parent as an errno. This replaces the + planned exception pipe. +- Create pipes with `pipe2(O_CLOEXEC)`. `dup2` onto fds 0, 1 and 2 clears + close-on-exec on the target fd, so the child still gets its stdio. + +glibc implements `posix_spawn` with `clone(CLONE_VM|CLONE_VFORK)`, which is +safe in a multithreaded process. Wrapping the spawn in `runInBoundThread` +(`createProc''` in the debug commit) does not help, because the other +threads still exist at fork time. + +## Build status at the time of the investigation + +- `use-native` was rebased onto master as one commit, `Add debug logging`. +- `flake.nix` points `streamly-packages` at revision `69728978…`, the + revision used by streamly's flake. This provides prebuilt + streamly-0.11.0 and streamly-core-0.3.0. +- The `-f use-native` build fails: + - The debug code does not compile against streamly-core 0.3, because the + `Dir.readFiles` API changed. + - Without the debug code, the native path on master does not compile + either. `standalone` uses `withCreateProcess` and the `std_in` field, + but the native `Config` is only a `Bool`. + +Because of these build failures, the test suite was not run with +`use-native`. The findings above come from the two small programs, not +from streamly-process itself. diff --git a/src/Streamly/Internal/System/Process.hs b/src/Streamly/Internal/System/Process.hs index 7611dafb..b01de68d 100644 --- a/src/Streamly/Internal/System/Process.hs +++ b/src/Streamly/Internal/System/Process.hs @@ -128,7 +128,7 @@ module Streamly.Internal.System.Process ) where -import Control.Concurrent (forkIO) +import Control.Concurrent (forkIO, runInBoundThread) import Control.Exception (Exception(..), catch, throwIO) import Control.Monad (void, unless) import Control.Monad.Catch (MonadCatch, throwM) @@ -616,6 +616,14 @@ createProc' modCfg path args = do Config cfg = modCfg $ mkConfig path args +createProc'' :: + (Config -> Config) -- ^ Process attribute modifier + -> FilePath -- ^ Executable path + -> [String] -- ^ Arguments + -> IO (Maybe Handle, Maybe Handle, Maybe Handle, ProcessHandle) +createProc'' modCfg path args = + runInBoundThread $ createProc' modCfg path args + {-# INLINE putChunksClose #-} putChunksClose :: MonadIO m => Handle -> Stream m (Array Word8) -> Stream m a diff --git a/src/Streamly/Internal/System/Process/Posix.hs b/src/Streamly/Internal/System/Process/Posix.hs index edff2707..fe87360a 100644 --- a/src/Streamly/Internal/System/Process/Posix.hs +++ b/src/Streamly/Internal/System/Process/Posix.hs @@ -1,4 +1,4 @@ -{-# LANGUAGE Safe #-} +-- {-# LANGUAGE Safe #-} -- | -- Module : Streamly.Internal.System.Process.Posix -- Copyright : (c) 2020 Composewell Technologies @@ -35,16 +35,19 @@ import Data.Tuple (swap) import GHC.IO.Device (IODeviceType(..)) import GHC.IO.Encoding (getLocaleEncoding) import GHC.IO.Handle.FD (mkHandleFromFD) -import System.IO (IOMode(..), Handle) +import System.IO (IOMode(..), Handle, hPutStrLn, stderr) import System.IO.Error (isDoesNotExistError) import System.Posix.IO (createPipe, dupTo, closeFd) -import System.Posix.Process (forkProcess, executeFile, ProcessStatus) +import System.Posix.Process (forkProcess, executeFile, ProcessStatus, getProcessID) import System.Posix.Types (ProcessID, Fd(..), CDev, CIno) import System.Posix.Signals (signalProcess, sigTERM) import System.Posix.Internals (fdGetMode) +import qualified Streamly.Internal.FileSystem.Dir as Dir +import qualified Streamly.Prelude as Stream import qualified GHC.IO.FD as FD import qualified System.Posix.Process as Posix +import Data.List (intercalate) ------------------------------------------------------------------------------- -- Utilities to create stdio handles @@ -54,13 +57,13 @@ import qualified System.Posix.Process as Posix -- We have to put the FDs into binary mode on Windows to avoid the newline -- translation that the CRT IO library does. setBinaryMode :: FD.FD -> IO () -#if defined(mingw32_HOST_OS) -setBinaryMode fd = do - _ <- setmode (FD.fdFD fd) True - return () -#else + + + + + setBinaryMode _ = return () -#endif + -- See Posix.fdToHandle and GHC.IO.Handle.FD.fdToHandle -- See stdin, stdout, stderr in module GHC.IO.Handle.FD @@ -112,9 +115,18 @@ mkPipeDupChild :: Direction -> Fd -> IO (Fd, (IO (), IO (), IO ())) mkPipeDupChild direction childFd = do let setDirection = if direction == ParentToChild then id else swap (child, parent) <- fmap setDirection createPipe + pid <- getProcessID let parentAction = closeFd child childAction = - closeFd parent >> void (dupTo child childFd) >> closeFd child + hPutStrLn stderr ("closing parent fd" ++ show parent) >> + closeFd parent >> + hPutStrLn stderr ("closed parent fd" ++ show parent) >> + hPutStrLn stderr ("duplicating child to fd" ++ show (child, childFd)) >> + void (dupTo child childFd) >> + hPutStrLn stderr ("duplicated child to fd" ++ show (child, childFd)) >> + hPutStrLn stderr ("closing child" ++ show child) >> + closeFd child >> + hPutStrLn stderr ("closed child" ++ show child) failureAction = closeFd child >> closeFd parent return (parent, (parentAction, childAction, failureAction)) @@ -144,7 +156,15 @@ mkStdioPipes pipeStdErr = do -} let parentAction = inpParent >> outParent >> errParent -- >> excParent - childAction = inpChild >> outChild >> errChild -- >> excChild + childAction = + hPutStrLn stderr "child input action doing" + >> inpChild + >> hPutStrLn stderr "child input action done" + >> hPutStrLn stderr "child output action doing" + >> outChild + >> hPutStrLn stderr "child output action done" + >> errChild -- >> excChild + -- childAction = inpChild >> outChild >> errChild -- >> excChild failureAction = inpFail >> outFail >> errFail -- >> excFail inpH <- toHandle WriteMode inp @@ -238,6 +258,7 @@ newProcess :: -> IO Process newProcess action path args env = do pid <- forkProcess exec + hPutStrLn stderr ("parent process " ++ show pid) pidToProcess pid Nothing where @@ -246,7 +267,12 @@ newProcess action path args env = do -- to the parent and clean up the parent fds. We can send the exceptions -- via a pipe like we do for threads. -- - exec = action >> executeFile path True args env + exec = do + pid <- getProcessID + hPutStrLn stderr ("child process " ++ show pid) + fds <- Stream.toList . Stream.unfold Dir.readFiles $ ("/proc/" ++ show pid ++ "/fd") + hPutStrLn stderr (intercalate ", " fds) + action >> executeFile path True args env newtype ProcessDoesNotExist = ProcessDoesNotExist ProcessID deriving Show