high reasource use on no overcommit systems

[nevakrien]

so i am seeing there is a use of fork on posix. I was wondering if it can be replaced to vfork since then you are not copying a memory page which seems just straight up better.

[Neko-Box-Coder]

Reading from the documentation, vfork() suspends the parent process until the child is finished. This is certainly not desirable given an async version is supported, not to mention it probably won't work if the system does follow the standard and suspends the parent.

Alternatively, I can do clone() with the CLONE_VM flag but it seems like clone() doesn't exist on MacOS.

Another one is posix_spawn() which seems to use clone() with CLONE_VM under the hood. This function should exist in MacOS but the MacOS will probably still use fork() which inherits the parent process memory.

Using posix_spawn() instead of fork() should solve this problem on Linux. Unfortunately there's no solution for this problem on MacOS. See here

[nevakrien]

mmmm that's tricky one solution could be break into a separate thread and work from there. like make a thread that runs vfork. (the suspend apears to be thread only https://man7.org/linux/man-pages/man2/vfork.2.html)
the issue is really only there for linux meant to run on servers so the alternative solution is to just posix_spawn like you mentioned or ifdef with clone on linux.

I will fixup my pull request to use clone with a flag on systems that have it since this seems like the least invasive

ig a separate solution for app devlopers could be calling vfork before doing all the command line stuff which would mean a much smaller memory pool to get COWed. I might end up going that route if we cant find a good solution (or i will remove async).

[Neko-Box-Coder]

Would posix_spawn() not work? (Since it uses clone() under the hood). I would prefer that over having a separate code path for Linux if possible.

I guess the ultimate solution for all posix is to fork at the beginning of the program and use that forked process to do fork() + exec()

But that will require a lot of work and restructuring.

[Neko-Box-Coder]

Sorry, forget what I said. Let's go with clone() so that Linux system is guaranteed to not inherit parent memory instead of relying on system glibc to be above 2.24, even if it means a separate code path for Linux.

[nevakrien]

seems like using clone requires alocating a stack which complicates things a lot actually.
I would try posix_spawn to avoid this. i do agree it sucks with the new glibc so what I am aiming for now is thread+vfork which should also work for apple. the idea is simply to inline the entire thing inside a sync function.

alternativly I could go the clone with mmap route and just relese the memory when the result from the process is gathered. this seems simple enough to excute

[Neko-Box-Coder]

Okay, yeah try posix_spawn().

vfork() won't work for apple because it does specifically say parent process instead of thread. See here

If not I will spend some time in the future to see if I can figure this out, although I can't guarantee any time frame at the moment.

[nevakrien]

seems posix_spawn wont quit work because it does not support changing dirs on linux (god like seriously????)
I can see a few solutions the easist of which is use vfork to quickly change dirs and then go back. not super pretty but it works.
we could also say that these specific comands are not supported in that mode because it is a feature flag

there is also changing directory without it or caving in and making a clone build on linux that alocates stack memory.
I need to look more into it but I think we can unmap the stack memory on the parent process to avoid needing to free it later.