Skip to content

bpo-45696: Deep-freeze#29118

Merged
gvanrossum merged 46 commits intopython:mainfrom
gvanrossum:deepfreeze
Nov 11, 2021
Merged

bpo-45696: Deep-freeze#29118
gvanrossum merged 46 commits intopython:mainfrom
gvanrossum:deepfreeze

Conversation

@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Oct 21, 2021 •
edited
Loading

See faster-cpython/ideas#84

This seems to be 10% faster than current main[1], so an improvement over Eric's freeze.

Downside: the data segment grows from 490 KB to 1.3 MB. (We could get ~300 KB back by not including the marshalled-frozen data.)

Another downside, for developers: whenever you touch a source file, _bootstrap_python gets rebuilt, and that triggers a rebuild of all the deep-frozen modules. (See comment below for a possible fix.)

Haven't gotten it working on Windows yet. (The magic is disabled there for now.) I might do that in a separate PR.

[1] Measured on an x86 Mac, with lto-pgo, this shaves 1.5-2 msec off a total run time of around 14 msec. I measured by using the subprocess to run python -c pass 100 times.

https://bugs.python.org/issue45696

ericsnowcurrently and others added 27 commits August 30, 2021 17:32
@ericsnowcurrently
Add placeholders for stdlib modules.
a04d702
@ericsnowcurrently
Add the frozen stdlib modules.
033acab
@ericsnowcurrently
Freeze the stdlib modules loaded during startup.
0591cbd
@ericsnowcurrently
Group the import-related configs.
72666d4
@ericsnowcurrently
Add the -X frozen_modules CLI option.
ce71894
@ericsnowcurrently
Ignore frozen modules depending on "-X frozen_modules".
ab500e2
@ericsnowcurrently
Remember the stdlib dir during startup.
932e563
@gvanrossum
First part of deepfreeze
2bb2a43 
Not generating Makefile yet
@ericsnowcurrently
Add _Py_GetMainConfig().
cb6c74d
@ericsnowcurrently
Let _Py_GetStdlibDir() fall back to a config.
e94240e
@ericsnowcurrently
Add _PyConfig_InitImportConfig().
e106e41
@ericsnowcurrently
Identify whether or not the executed is running installed.
80506bb
@ericsnowcurrently
Default to "off" if in development.
9ac5b11
@ericsnowcurrently
Remember if built with --enable-optimizations.
acfc2a2
@ericsnowcurrently
Default to "on" for PGO builds.
ef653fb
@gvanrossum
Make deepfreeze actually work
f70319b 
Still not generating Makefile
@ericsnowcurrently
Add a NEWS entry.
a381ccf
@gvanrossum
Generate all deepfreeze items in Makefile.pre.in
47e3917
@gvanrossum
Don't use GET_CODE in the extern declarations
20992e9
@gvanrossum
Merge branch 'pr-28107' into deepfreeze
692959b
@gvanrossum
Exclude encodings module for now
77812d6
@gvanrossum
Regenerate Python/frozen.c
80cf00f
@gvanrossum
Make [deep]freeze process mess noisy
1d29e95
@gvanrossum
Regenerated MSVC project files
9818c44
@gvanrossum
Merge branch 'main' into deepfreeze
9e817b2
@gvanrossum
Make deep-freezing work again
2d89d0a
@gvanrossum
Make deep-freezing *really* work
8d029ae
@gvanrossum
Copy link
Member Author

gvanrossum commented Nov 9, 2021

@tiran any idea?

@tiran
Copy link
Member

tiran commented Nov 9, 2021

@tiran any idea?

You could try to use the bootstrap interpreter to generate sysconfig and pybuilddir.txt: main...tiran:deepfreeze-bootstrap

In the long run we might be able to replace PYTHON_FOR_REGEN with bootstrap interpreter, too.

@gvanrossum
Copy link
Member Author

gvanrossum commented Nov 9, 2021

@tiran, Does it look good now? All tests pass. I'll merge origin/main again and re-run the buildbots and then I hope to land this. We'll leave several things to future generations (or at least a later PR):

  • Windows
  • Avoid deep-freezing and rebuilding after every source change
  • Replace PYTHON_FOR_REGEN with $(BOOTSTRAP)

@gvanrossum gvanrossum added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@bedevere-bot
Copy link

bedevere-bot commented Nov 10, 2021

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit 0c9de73 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@gvanrossum
Copy link
Member Author

gvanrossum commented Nov 10, 2021

The AMD64 FreeBSD buildbot is failing (https://buildbot.python.org/all/#/builders/203/builds/314/steps/3/logs/stdio), and AFAICT the cause is that its Make doesn't understand $<. Any suggestions? This part of Makefile.pre.in is generated code, so I could just expand it in the code generator (Tools/scripts/freeze_modules.py), like I had before. Or is there a trick to get this to work on FreeBSD?

@tiran
Copy link
Member

tiran commented Nov 10, 2021

I was not aware that bmake has a different syntax for autovars than gmake. I only develop with GNU make. I cannot find a common syntax that works with BSD make and GNU make. You have to go back to the old system.

I'm sorry for the extra work I caused.

@gvanrossum gvanrossum added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@bedevere-bot
Copy link

bedevere-bot commented Nov 10, 2021

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit 3e881031b14dc98f7d2b579e6b06c151df886333 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@gvanrossum gvanrossum added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@bedevere-bot
Copy link

bedevere-bot commented Nov 10, 2021

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit f40d138 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@gvanrossum gvanrossum merged commit 1cbaa50 into python:main Nov 11, 2021
@gvanrossum gvanrossum deleted the deepfreeze branch November 11, 2021 02:01
remykarem pushed a commit to remykarem/cpython that referenced this pull request Dec 7, 2021
@gvanrossum @remykarem
bpo-45696: Deep-freeze selected modules (pythonGH-29118)
e200b3e 
This gains 10% or more in startup time for `python -c pass` on UNIX-ish systems.

The Makefile.pre.in generating code builds on Eric's work for bpo-45020, but the .c file generator is new.

Windows version TBD.
remykarem pushed a commit to remykarem/cpython that referenced this pull request Jan 30, 2022
@gvanrossum @remykarem
bpo-45696: Deep-freeze selected modules (pythonGH-29118)
d5c35ec 
This gains 10% or more in startup time for `python -c pass` on UNIX-ish systems.

The Makefile.pre.in generating code builds on Eric's work for bpo-45020, but the .c file generator is new.

Windows version TBD.
@kumaraditya303 kumaraditya303 mentioned this pull request Feb 2, 2022
Merged
miss-islington pushed a commit that referenced this pull request Feb 4, 2022
@kumaraditya303
bpo-46608: exclude marshalled-frozen data if deep-freezing to save 30…
bf95ff9 
…0 KB space (GH-31074)

This reduces the size of the data segment by **300 KB** of the executable because if the modules are deep-frozen then the marshalled frozen data just wastes space. This was inspired by comment by @gvanrossum in #29118 (comment). Note: There is a new option `--deepfreeze-only` in `freeze_modules.py` to change this behavior, it is on be default to save disk space.
```console 
# du -s ./python before
27892   ./python
# du -s ./python after
27524   ./python
```

Automerge-Triggered-By: GH:ericsnowcurrently
@gvanrossum gvanrossum mentioned this pull request Apr 10, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@methane methane methane left review comments

@tiran tiran tiran left review comments

@ericsnowcurrently ericsnowcurrently ericsnowcurrently approved these changes

@encukou encukou Awaiting requested review from encukou

@ncoghlan ncoghlan Awaiting requested review from ncoghlan ncoghlan is a code owner

@warsaw warsaw Awaiting requested review from warsaw warsaw is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@gvanrossum @bedevere-bot @tiran @methane @ericsnowcurrently @the-knights-who-say-ni