Thanks @pablogsal for attacking this issue :-)

@pitrou Thank you very much for the review!

I have simplified the API. Now setting a custom reducer looks like this:

   import multiprocessing
   from multiprocessing.reduction import AbstractReducer, ForkingPickler

    class ForkingPicklerProtocol2(ForkingPickler):
       @classmethod
       def dumps(cls, obj, pickle_protocol=2):
           return super().dumps(obj, protocol=pickle_protocol)

    class PickleProtocol2Reducer(AbstractReducer):
       def get_pickler_class(self):
           return ForkingPicklerProtocol2

    multiprocessing.set_reducer(PickleProtocol2Reducer)

I am making the interface a bit more strict, so multiprocessing.set_reducer() must be called with a subclass of AbstractReducer and get_pickler_class must return a subclass of pickler.Pickle. This way, the constructor and the rest of the methods needed for the multiprocessing reduction machinery will be there. I have added some new test that check this behaviour.

@pablogsal Do you need another review on this?

@pitrou It took me a while but I have stabilized all tests and fixed some details on Windows. I have also added Listener and Client to the context so they also can benefit from custom reducers. Please, check my previous comment regarding some details.

This patch is already very big and very very complex and when errors happen they are extremely obscure or platform dependent, so I apologize in advance if I miss something obvious, but I have too many spinning plates.

Could you take another look?

@pitrou There is some failures in Windows that I am investigating but I found a problem.

In multiprocessing/context.py there is no way of passing down the current context to the Process class. The Process class is an attribute of DefaultContext. If we change that to a function that injects the context down, any class that is inheriting from multiprocessing.Process will be broken because now it would not be a class (is a function).

This is the only exception as any other class is defined in BaseContext as a method that injects the context down, but not the process. Passing the context to the process is necessary because it calls dump (check for example multiprocessing/popen_spawn_win32.Popen.init).

I don't know how to solve this, but basically tests like test_queue_custom_reducer_over_custom_context are not possible as there is no way for the call to Process to pass down the context unless you do something ugly as:

self.custom_ctx.Process(..., ctx=self.custom_ctx)

Trying to do something like:

class DefaultContext(BaseContext):
    @property
    def Process(self):
        class _Process(Process):
             _ctx = self.get_context()
        return _Process

fails because _Process will not be picklable downstream. So I am out of ideas.

As you are more used to the architecture of the multiprocessing module, do you see a way of solving this?

If you don't see a way, I'm afraid that custom reducers per context cannot be implemented because of the way multiprocessing is architected.

@pablogsal thanks a lot for putting this PR together, this is great work. This feature is very promising.

Regarding the issue concerning the mp.Process class, it is definitely tricky. I think we should leverage the fact that we only want this context injection into the process class when the user explicitly asks for custom reduction behavior.
As a result, we can make Process a property of Context objects, and make the property return a different thing depending on if the user asked for custom reduction, which is a new feature (no backward compat issue), or not.

See proof of concept below

    @property
    def Process(self):
        if not self._custom_reduction_enabled:
            # Ensure backward compatibility by returning a class when no
            # custom reducer was specified
            return _Process  # ForkProcess for ForkContext, Process for BaseContext etc.
        else:
            return self.process_factory

    def process_factory(self, *args, **kwargs):
        p = Process(*args, **kwargs)
        p._ctx = self.get_context()
        return p

More complete implementation here:
https://github.com/pablogsal/cpython/compare/bpo28053...pierreglaser:inject-custom-context-into-process-cls?expand=1

Gist showing a usage example and its behavior:
https://gist.github.com/pierreglaser/ed4f9f9e784a3571cfdc8b969d32085f

What do you think?

EDIT: Another question is whether or not we consider that Context custom classes are part of the public API. The only place I think where Context classes are mentioned in the python docs is there:

If we want to enable set_reducer with custom Context classes that have a Process attribute, most probably pointing to a Process class, then my above snippet will break -- how to handle this case is still an open question.

Another topic that appears many times in this PR is the strange AbstractPickler.loads classmethod and the symmetry between the load and dump for the Pickler subclasses returned by get_pickler_class.

A good way IMO to re-establish the symmetry here would be to use actual Unpickler classes to load pickle strings, instead of using smoke load methods of Pickler subclasses.

Thus, we could add an optional get_unpickler_class to the AbstractReducer API. The returned class would have to implement a load instance method. This way, the symmetry is re-established, and we do not create un-necessary Pickler instances at load time.