Our current organization of pytest configuration files is not optimal for locally running tests and could be improved to add flexibility and increase utility for both local and CI runs. Most of what follows are my personal issue with the current setup and ideas for how it could be improved. I would love to have a discussion on this topic and incorporate feedback from the team.

For some context, I first started looking at this when trying to use the --run_stress commands locally and ran into issues:

~/Repos/rapids/cuml-dev2/python$ pytest --run_stress
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --run_stress
  inifile: /home/mdemoret/Repos/rapids/cuml-dev2/python/pytest.ini
  rootdir: /home/mdemoret/Repos/rapids/cuml-dev2/python

The way we currently have our repo organized is not recommended by pytest. They suggest moving the conftest.py folder to the root of the python package or to create plugins since conftest.py deeper in directories are not scanned initially. Given the growing nature of our current conftest.py, growing number of tests, and need to adjust which tests are run, I think it makes sense to make the following changes:

1. Move all command line options added in pytest_addoption to plugins in python/cuml/test/plugins
2. Require the necessary plugins using the addopts section in pytest.ini (i.e. addopts = -p "cuml.test.plugins.custom_options_plugin")
   1. And in general, make better use of pytest.ini and rely less on command line arguments in ci/gpu/build.sh
3. Change the --import-mode to --import-mode=importlib
   1. This is necessary to make addopts = -p "cuml.test.plugins.custom_options_plugin" work and will become the default. This also makes it easier during development since both local and pip installed packages can be resolved.
4. Move optional functionality in conftest.py to new plugins
   1. For example, the bad_cuml_array_check could be a plugin which would make it disable-able
5. Increase our use of markers.
   1. Markers are a great way to pick and choose which tests to skip or focus on. We don't use them enough

Making these changes will allow us more flexibility when running pytest locally and in CI. I have already tested some of these out and they work well. For example, I added a memory checker plugin here, and also added a "quick_run" plugin here which reduces the number of parameters run for each test (overall reduction from 34267 test to 2184).

Would be interested to hear everyone else's thoughts on the matter