Added: Possibility to specify extension modules. by margru · Pull Request #640 · pybuilder/pybuilder

margru · 2019-08-08T09:59:35Z

This is an improvement to fix #639. The extensions might be defined in project's distutils_ext_modules property as a list of dictionaries where each dictionary represents keyword arguments with values for a single Extension instance constructor. Multiple dictionaries mean multiple extensions to build. The extensions specifications are then rendered into the setup.py script via the template.

esc · 2019-08-08T11:37:22Z

@margru thank you for you pull-request, we do appreciate it. Unfortunately, we don't have any resources at the moment to service it so please be patient.

Adapted: Some integration tests to pass with extension modules parameter.

arcivanov · 2019-08-28T17:20:54Z

Thanks for the PR! I'm working on a major fix for the PyB with environment isolation and will look at this ASAP afterwards.

Python sources of cythonized modules can be removed from distribution.

margru · 2019-11-04T09:24:47Z

this makes no sense ;)

What doesn't make sense? Can you please be more specific?

Edit: Sorry, I didn't notice that you are commenting just a specific line in the code. See my comment there...

coveralls · 2019-11-04T09:27:29Z

Pull Request Test Coverage Report for Build 11064057482

Details

76 of 91 (83.52%) changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.06%) to 82.551%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/main/python/pybuilder/plugins/python/distutils_plugin.py	72	87	82.76%

Totals
Change from base Build 10380100976:	-0.06%
Covered Lines:	5540
Relevant Lines:	6493

💛 - Coveralls

Before this, the options provided in the requirements.txt file appear in the install_requires field which then causes errors (e.g. when using cython extensions).

Change default value when ext_modules are empty and update to upstream v0.13.5 and

arcivanov · 2022-08-03T14:12:43Z

Could you please rebase on current?

arcivanov · 2022-08-05T23:27:17Z

@margru ?

margru · 2022-08-08T05:18:04Z

@arcivanov Yeah, I will try but I don't have that much time right now to solve the conflicts. I didn't realize that the push will also update the PR...

Merge upstream to our patched code

Add Cython as a build dependency to setup.py

Add support for cython's compiler_directives

It contains necessary build requirements (currently only Cython if cythonization is required).

(cherry picked from commit 0cc08a8)

It contains necessary build requirements (currently only Cython if cythonization is required). (cherry picked from commit 9565af3)

# Conflicts: # src/main/python/pybuilder/plugins/python/distutils_plugin.py

Fix incorrect merge

# Conflicts: # src/main/python/pybuilder/plugins/python/unittest_plugin.py

- Maybe related only to Win platform.

arcivanov · 2024-05-29T21:49:05Z

src/main/python/pybuilder/plugins/python/vendorize_plugin.py

                continue

-            with open(lic_file, "rt") as f:
+            with open(lic_file, "rt", encoding="utf8") as f:


That's really not necessary and works everywhere.

If it worked everywhere, I wouldn't have to do this change ;) The original code was raising an error for me - UnicodeDecodeError regarding 'charmap' encoding...

It doesn't do it in pristine test build environments though, including Windows on which we test. You can introduce terrible and intractable errors by forcing non-local file encoding the way you're doing it here.

Observe: https://github.com/pybuilder/pybuilder/actions/runs/9262689796/job/25480202283#step:3:1353

OK, but do you have any idea why it's raising the UnicodeDecode error in my environment? Or what settings should I check to get rid of it? Otherwise it's useless to know that it's working in the pristine build env if I cannot build it in my env... Of course, I tried to recreate the virtualenv I am using for PyBuilder development, still the same... I am working on Win64, Python 3.10.

There is no "Win64", except for a platform name, which doesn't say much - the CI tests also run on "Win64" and they are fine.

What version of Windows EXACTLY are you using?
Can you please run python -c 'import sys; print(sys.getdefaultencoding())' and post the output?
Can you please try using PYTHONIOENCODING=utf8 and run your pyb with this env var on and report whether it made a difference (you'd need to remove your encoding patches from your branch first though to see the effect )?

Where did you get your Python from?

the output of the command: ascii

This isn't normal AT ALL. Windows is usually utf-16 (LE IIRC). But, more importantly, Python has been defaulting to UTF-8 since 3.6 for IO.

https://peps.python.org/pep-0528/
https://peps.python.org/pep-0529/
https://peps.python.org/pep-0540/
https://docs.python.org/3/using/windows.html#utf-8-mode
https://docs.python.org/3/library/os.html#utf8-mode

Well, my fault, I launched the command in the default Python interpreter which is Python27 - I have multiple Python versions installed on my system. However, I have a virtualenv for building Pybuilder with Python310 and when I launch the command there, I get utf-8 even without setting the envvar. That, I think, makes the situation even more weird...

What's your source for Python? ActivePython?

No, all my Python versions downloaded from https://www.python.org/.

? Any new hint on what to try?

margru

This merge caused that building with PyBuilder is broken:

Step #2 - "build_package_3.10": Traceback (most recent call last):
Step #2 - "build_package_3.10":   File "/usr/local/bin/pyb", line 5, in <module>
Step #2 - "build_package_3.10":     from pybuilder.cli import main
Step #2 - "build_package_3.10":   File "/usr/local/lib/python3.10/site-packages/pybuilder/cli.py", line 37, in <module>
Step #2 - "build_package_3.10":     from pybuilder.reactor import Reactor
Step #2 - "build_package_3.10":   File "/usr/local/lib/python3.10/site-packages/pybuilder/reactor.py", line 37, in <module>
Step #2 - "build_package_3.10":     from pybuilder.pluginloader import (BuiltinPluginLoader,
Step #2 - "build_package_3.10":   File "/usr/local/lib/python3.10/site-packages/pybuilder/pluginloader.py", line 33, in <module>
Step #2 - "build_package_3.10":     from pybuilder.pip_common import Version
Step #2 - "build_package_3.10":   File "/usr/local/lib/python3.10/site-packages/pybuilder/pip_common.py", line 20, in <module>
Step #2 - "build_package_3.10":     import pkg_resources
Step #2 - "build_package_3.10":   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
Step #2 - "build_package_3.10":   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
Step #2 - "build_package_3.10":   File "<frozen importlib._bootstrap>", line 672, in _load_unlocked
Step #2 - "build_package_3.10":   File "<frozen importlib._bootstrap>", line 632, in _load_backward_compatible
Step #2 - "build_package_3.10":   File "/usr/local/lib/python3.10/site-packages/pybuilder/extern/__init__.py", line 74, in load_module
Step #2 - "build_package_3.10":     mod = import_module(extant)
Step #2 - "build_package_3.10":   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
Step #2 - "build_package_3.10":     return _bootstrap._gcd_import(name[level:], package, level)
Step #2 - "build_package_3.10":   File "/usr/local/lib/python3.10/site-packages/pybuilder/_vendor/pkg_resources/__init__.py", line 96, in <module>
Step #2 - "build_package_3.10":     from ..jaraco.text import (
Step #2 - "build_package_3.10":   File "/usr/local/lib/python3.10/site-packages/pybuilder/_vendor/jaraco/text/__init__.py", line 233, in <module>
Step #2 - "build_package_3.10":     files(__name__).joinpath('Lorem ipsum.txt').read_text(encoding='utf-8')
Step #2 - "build_package_3.10":   File "/usr/local/lib/python3.10/pathlib.py", line 1134, in read_text
Step #2 - "build_package_3.10":     with self.open(mode='r', encoding=encoding, errors=errors) as f:
Step #2 - "build_package_3.10":   File "/usr/local/lib/python3.10/pathlib.py", line 1119, in open
Step #2 - "build_package_3.10":     return self._accessor.open(self, mode, buffering, encoding, errors,
Step #2 - "build_package_3.10": FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/site-packages/pybuilder/_vendor/jaraco/text/Lorem ipsum.txt'
Step #2 - "build_package_3.10": Incorrect pybuilder version, expecting v0.13

arcivanov · 2024-09-27T16:41:59Z

I dont think so - look at the local builds.

margru · 2024-09-27T16:50:57Z

I didn't mean building PyBuilder, but building with PyBuilder. When I build PyBuilder based on this commit, it builds OK. But when I use that PyBuilder version to build another package, I get the mentioned error (that I don't get when using the previous commit).

EDIT: Just a note: I have set the PyBuilder version to 0.13.99.dev for my custom PyBuilder builds so that it doesn't interfere with the original PyBuilder. I don't know, if that might break something but it has been working OK for me so far...

arcivanov · 2024-09-27T17:37:24Z

I didn't mean building PyBuilder, but building with PyBuilder.

Ah! A "couple of two" things 😄

I'm always worried of the things installed in /usr because it's a system thing and who knows how it behaves.
Integration tests use PyBuilder to build projects, i.e. this scenario should be covered.

Let me see if I can reproduce it locally with master.

Added: Possibility to specify extension modules.

8ed363c

margru added 3 commits August 8, 2019 14:54

Added: Integration test for the extension module functionality.

d044794

Adapted: Some integration tests to pass with extension modules parameter.

Adapted: Some unit tests to pass with extension modules parameter.

c6e9e45

Fixed: Code-style breaking the build.

2b58c96

margru force-pushed the master branch from 9c60a2a to 2b58c96 Compare August 9, 2019 13:44

margru added 5 commits August 27, 2019 16:46

Added possibility to build Cython extensions

842d1c0

Fixed Cython extension rendering in setup.py

497ba24

Minor fixes

593690f

Unpinned pip and setuptools dependencies

c20e39b

Merge branch 'dev'

e1d9387

Improved Cython extension modules handling

8dde5e6

Python sources of cythonized modules can be removed from distribution.

margru and others added 7 commits March 11, 2020 19:02

Added support for include_package_data

5ef5f67

Minor fixes

ac944e9

Discard pip options from the requirements file

5dcc1f8

Before this, the options provided in the requirements.txt file appear in the install_requires field which then causes errors (e.g. when using cython extensions).

Merge branch 'margu/master' into 'upstream/master'

a94bf4a

Change default value when ext_modules are empty

168a57a

Fix Cython dependency for cython_ext_modules

e7ccffb

Merge pull request #1 from chylek/ext_modules

301ef9b

Change default value when ext_modules are empty and update to upstream v0.13.5 and

chylek added 5 commits August 23, 2022 13:29

Merge remote-tracking branch 'upstream/master' into ext_modules

3d9f6d7

Fix ext modules test

6c96c06

Improve Cython dependency check

d6f6f65

Fix ext modules test

5a50a10

Add lazy Cython import in setup.py

9380b35

margru and others added 23 commits June 16, 2023 12:03

Merge pull request #3 from chylek/update_upstream

f45b8db

Merge upstream to our patched code

Merge pull request #4 from chylek/cython_in_setup

45bf3ba

Add Cython as a build dependency to setup.py

Merge branch 'master' into feature/compiler_directives

95cbd38

Merge pull request #5 from chylek/feature/compiler_directives

b70a141

Add support for cython's compiler_directives

Add necessary setup_requires

0cc08a8

FIX integration tests

2d071d7

Minor fix

809442d

Add pyproject.toml creation if not exist

9565af3

It contains necessary build requirements (currently only Cython if cythonization is required).

Merge remote-tracking branch 'origin/ext_modules' into master-fix

d01363d

Merge remote-tracking branch 'origin/cython_in_setup' into master-fix

ec6fe88

Add support for cython's compiler_directives

18cf168

Add necessary setup_requires

0164015

(cherry picked from commit 0cc08a8)

Add pyproject.toml creation if not exist

d1470c4

It contains necessary build requirements (currently only Cython if cythonization is required). (cherry picked from commit 9565af3)

Add pyproject.toml merging

c4b168e

Fix linting issue

0e26825

Fix lazy cythonization

71722fe

Merge remote-tracking branch 'chylek/master-fixed'

2c35dfa

# Conflicts: # src/main/python/pybuilder/plugins/python/distutils_plugin.py

Fix incorrect merge

46bea10

Merge pull request #6 from chylek/master

dc444d8

Fix incorrect merge

Merge tag 'v0.13.11'

41b6792

# Conflicts: # src/main/python/pybuilder/plugins/python/unittest_plugin.py

Merge branch 'master' into master

05aa208

Minor fix causing error during build

dab1233

- Maybe related only to Win platform.

FIX cython version for cythonization for > 3.8

0fe2c92

arcivanov reviewed May 29, 2024

View reviewed changes

margru and others added 2 commits September 26, 2024 16:41

FIX cython versions setting

981b13b

Merge branch 'master' into master

443cd45

margru commented Sep 27, 2024

View reviewed changes

Conversation

margru commented Aug 8, 2019

Uh oh!

esc commented Aug 8, 2019

Uh oh!

arcivanov commented Aug 28, 2019

Uh oh!

margru commented Nov 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Nov 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 11064057482

Details

💛 - Coveralls

Uh oh!

arcivanov commented Aug 3, 2022

Uh oh!

arcivanov commented Aug 5, 2022

Uh oh!

margru commented Aug 8, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arcivanov May 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

margru left a comment

Choose a reason for hiding this comment

Uh oh!

arcivanov commented Sep 27, 2024

Uh oh!

margru commented Sep 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arcivanov commented Sep 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

margru commented Nov 4, 2019 •

edited

Loading

coveralls commented Nov 4, 2019 •

edited

Loading

arcivanov May 30, 2024 •

edited

Loading

margru commented Sep 27, 2024 •

edited

Loading