feat: process isolation and fork-safe test runners

[nicklafleur]

Pluggable process isolation, with a fork-safe hot-fork runner

Adds a MutantRunner abstraction that owns the process-isolation strategy for mutation testing

Two strategies:

• fork (default) the existing fork-per-mutant behaviour, extracted into ForkRunner.
  No behavioural change.
• hot-fork parent (clean) -> orchestrator (imports pytest once) -> N grandchildren,
  for projects whose test setup makes the parent fork-unsafe (gevent monkey-patching,
  grpc, torch). The parent never imports pytest or conftest; the orchestrator imports it
  exactly once and forks a grandchild per mutant. Results stream back over a
  length-prefixed pipe protocol, and grandchildren are reaped via a SIGCHLD self-pipe.
  A crashed orchestrator is restarted with its in-flight mutants re-submitted, up to
  max_orchestrator_restarts, before raising OrchestratorCrashError.

Stats collection, clean tests, forced-fail and test listing all run in short-lived forks
(run_in_fork / run_in_fork_with_result) so the parent stays clean throughout.

Config

Option	Default	Purpose
process_isolation	fork	fork or hot-fork
hot_fork_warmup	collect	collect / import / none
max_orchestrator_restarts	3	Restarts before OrchestratorCrashError
preload_modules_file	None	Modules to preload in the orchestrator
log_to_file	False	Enable orchestrator file logging
log_file_path	mutants/mutmut-debug.log	Log destination

Incidental cleanups

Breaks the mutation/trampoline.py -> __main__ import cycle: MutmutProgrammaticFailException
moves to core.py and record_trampoline_hit to stats.py, which removes the
_set_mutant_under_test late-import shim. src/ now has zero from mutmut.__main__
imports. Also drops the dead pid-keyed bookkeeping from SourceFileMutationData
(register_pid / register_result / stop_children), superseded by MutantResult, and
sets platform = linux in mypy.ini, mutmut is (currently) POSIX-only, so type-checking
elsewhere otherwise flags os.fork / resource.setrlimit / signal.SIGXCPU as missing.

Testing

Verified that fork and hot-fork produce identical verdicts on the my_lib e2e project.
New coverage for the pipe protocol (including payloads larger than the pipe buffer),
ForkRunner/HotForkRunner no-test-mutant handling, get_mutant_runner dispatch,
OrchestratorCrashError messaging, and config validation, plus a subprocess e2e smoke
test (tests/e2e/test_e2e_hot_fork.py with the hot_fork_basic project) exercising the
full hot-fork path.

[Otto-AA]

Thanks for the PR ❤️

I'll try to look a bit more at it today or tomorrow.

From a high level, I think we get the following (?):

Does it ...	fork	hot-fork
Cache all imports? (pytest and mutated source code)	Yes (we ran the whole pytest suite)	Yes (configurable via hot_fork_warmup, either pytest --collect-only or fixed list of imports or nothing)
Cache session fixtures	Not sure actually, whether executing pytest twice from python code will call the session fixture twice.	No
Run pytest multiple times within the same process?	Yes	No (except with mutate_only_covered_lines ?)

I think the hot-fork + mutate_only_covered_lines combination is mabye not as expected: We run pytest via gather_coverage first in the main process and later on fork the orchestrator with pytest already being loaded. Should we also run the coverage collection via the

What this PR would fix:

• Fixes Fork-per-mutant inherits the parent's dead state; async tests are recorded as timeouts instead of their verdicts #578 (with the hot-fork option, because nothing "process-global" is created before forking)

What this PR nearly fixes:

• Fixes mutate_only_covered_lines crashes on C-extension deps (numpy, libyaml): cannot load module more than once per process #528 (with the hot-fork option. But I think because we run coverage collection in the main process and then fork afterwards in startup, we still have the same problem)

What this PR does not attempt to fix:

• Support parallel tests with external database #474 (at least in my use case, we use session fixtures to setup database tables and connections. I think session fixtures are not usable in the hot-fork scenario)

[Otto-AA]

I won't have time to review the code in the next weeks, so here are some high level inputs (feel free to incorporate or to merge without changes):

• the process_isolation and hot_fork_warmup should be explained in the README. As a user, I'd like to know when to use which of those.
• I feel like the naming fork and hot-fork does not reflect the differences between those two. I'm not sure how to better name them though
• I feel like it could fix #528, but currently does not. I haven't verified it though. (could also be fixed in a separate PR)

[Otto-AA]

Is there a reason you use a different setup than for the other tests?

This seems to work:

def test_hot_fork_runs_end_to_end():
    assert run_mutmut_on_project("hot_fork_basic") == snapshot(
        {
            "mutants/src/hf_calc/__init__.py.meta": {
                "hf_calc.x_add__mutmut_1": 1,
                "hf_calc.x_sub__mutmut_1": 1,
                "hf_calc.x_mul__mutmut_1": 1,
                "hf_calc.x_untested__mutmut_1": 33,
                "hf_calc.x_untested__mutmut_2": 33,
            }
        }
    )

[nicklafleur]

I believe this branch predated adding the snapshot to the project and I must have just gone blind to it. nice catch!

[nicklafleur]

I won't have time to review the code in the next weeks, so here are some high level inputs (feel free to incorporate or to merge without changes):

• the process_isolation and hot_fork_warmup should be explained in the README. As a user, I'd like to know when to use which of those.
• I feel like the naming fork and hot-fork does not reflect the differences between those two. I'm not sure how to better name them though
• I feel like it could fix mutate_only_covered_lines crashes on C-extension deps (numpy, libyaml): cannot load module more than once per process #528, but currently does not. I haven't verified it though. (could also be fixed in a separate PR)

good call on these suggestions, especially 528. I'll see if I can slot these in neatly. I agree hot-fork is not the best name, I named it based on the idea that the fork gets warmed, but I should find something better. I'll see what claude can come up with, definitely has better ideas than I can come up with for naming things.