Skip to content

bpo-45696: Deep-freeze #29118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gvanrossum merged 46 commits into from
Nov 11, 2021
Merged

bpo-45696: Deep-freeze #29118

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
a04d702
Add placeholders for stdlib modules.
ericsnowcurrently Aug 12, 2021
033acab
Add the frozen stdlib modules.
ericsnowcurrently Jun 8, 2021
0591cbd
Freeze the stdlib modules loaded during startup.
ericsnowcurrently Jun 8, 2021
72666d4
Group the import-related configs.
ericsnowcurrently Aug 31, 2021
ce71894
Add the -X frozen_modules CLI option.
ericsnowcurrently Aug 31, 2021
ab500e2
Ignore frozen modules depending on "-X frozen_modules".
ericsnowcurrently Aug 31, 2021
932e563
Remember the stdlib dir during startup.
ericsnowcurrently Jun 12, 2021
2bb2a43
First part of deepfreeze
gvanrossum Aug 31, 2021
cb6c74d
Add _Py_GetMainConfig().
ericsnowcurrently Aug 31, 2021
e94240e
Let _Py_GetStdlibDir() fall back to a config.
ericsnowcurrently Aug 31, 2021
e106e41
Add _PyConfig_InitImportConfig().
ericsnowcurrently Aug 31, 2021
80506bb
Identify whether or not the executed is running installed.
ericsnowcurrently Aug 31, 2021
9ac5b11
Default to "off" if in development.
ericsnowcurrently Aug 31, 2021
acfc2a2
Remember if built with --enable-optimizations.
ericsnowcurrently Aug 31, 2021
ef653fb
Default to "on" for PGO builds.
ericsnowcurrently Aug 31, 2021
f70319b
Make deepfreeze actually work
gvanrossum Aug 31, 2021
a381ccf
Add a NEWS entry.
ericsnowcurrently Aug 31, 2021
47e3917
Generate all deepfreeze items in Makefile.pre.in
gvanrossum Sep 1, 2021
20992e9
Don't use GET_CODE in the extern declarations
gvanrossum Sep 1, 2021
692959b
Merge branch 'pr-28107' into deepfreeze
gvanrossum Sep 1, 2021
77812d6
Exclude encodings module for now
gvanrossum Sep 1, 2021
80cf00f
Regenerate Python/frozen.c
gvanrossum Sep 5, 2021
1d29e95
Make [deep]freeze process mess noisy
gvanrossum Sep 5, 2021
9818c44
Regenerated MSVC project files
gvanrossum Sep 6, 2021
9e817b2
Merge branch 'main' into deepfreeze
gvanrossum Oct 20, 2021
2d89d0a
Make deep-freezing work again
gvanrossum Oct 20, 2021
8d029ae
Make deep-freezing *really* work
gvanrossum Oct 21, 2021
27518ff
Merge branch 'main' into deepfreeze
gvanrossum Oct 21, 2021
5e98069
Fix test_ctypes and struct_frozen example in docs
gvanrossum Oct 21, 2021
239ee0b
Merge branch 'main' into deepfreeze
gvanrossum Nov 1, 2021
34b6da4
Undo some badly merged files
gvanrossum Nov 1, 2021
2c54cf0
Add dependency on bootstrap for _bootstrap_python
gvanrossum Nov 1, 2021
502b566
Remove Programs/_bootstrap_python in 'make clean'
gvanrossum Nov 2, 2021
6733acd
Move _bootstrap_python to toplevel dir, to get build working
gvanrossum Nov 2, 2021
70d10f8
Regenerate Makefile.pre.in
gvanrossum Nov 2, 2021
11182f8
Use 'make' shortcuts in deep-freeze rules
gvanrossum Nov 2, 2021
b4103cb
Fix test_import, test_regrtest, test_trace
gvanrossum Nov 2, 2021
b7abc3e
Fix out-of-tree builds
gvanrossum Nov 2, 2021
6e11529
📜🤖 Added by blurb_it.
blurb-it[bot] Nov 3, 2021
fa52af4
Open files in UTF-8 mode
gvanrossum Nov 4, 2021
e4f31d0
Solve out-of-tree build differently
gvanrossum Nov 9, 2021
76fb82d
Use _bootstrap_python to create pybuilddir.txt and sysconfig
gvanrossum Nov 9, 2021
cb69845
Remove Python/deepfreeze contents
gvanrossum Nov 9, 2021
0c9de73
Merge remote-tracking branch 'origin/main' into deepfreeze
gvanrossum Nov 9, 2021
c747a0f
Don't use $@ and $<, they don't work on FreeBSD
gvanrossum Nov 10, 2021
f40d138
Fix out of tree builds by adding $(srcdir)
gvanrossum Nov 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'main' into deepfreeze
  • Loading branch information
@gvanrossum
gvanrossum committed Oct 20, 2021
commit 9e817b28320e15c740803cc92ae1d051c8ebd16f
3 changes: 1 addition & 2 deletions Doc/using/cmdline.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -484,8 +484,7 @@ Miscellaneous options
ignored by the import machinery. A value of "on" means they get
imported and "off" means they are ignored. The default is "on"
if this is an installed Python (the normal case). If it's under
development (running from the build dir) then the default is "off".
(A PGO build always defaults to "on", even if under development.)
development (running from the source tree) then the default is "off".
Note that the "importlib_bootstrap" and "importlib_bootstrap_external"
frozen modules are always used, even if this flag is set to "off".

Expand Down
3 changes: 1 addition & 2 deletions Include/internal/pycore_pylifecycle.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,7 @@ PyAPI_FUNC(PyStatus) _Py_PreInitializeFromConfig(
const PyConfig *config,
const struct _PyArgv *args);

PyAPI_FUNC(const wchar_t *) _Py_GetStdlibDir(const PyConfig *);
PyAPI_FUNC(bool) _Py_IsInstalled(const PyConfig *);
PyAPI_FUNC(wchar_t *) _Py_GetStdlibDir(void);

PyAPI_FUNC(int) _Py_HandleSystemExit(int *exitcode_p);

Expand Down
108 changes: 1 addition & 107 deletions Makefile.pre.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1013,94 +1013,6 @@ regen-frozen: Tools/scripts/freeze_modules.py $(FROZEN_FILES_IN)
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/scripts/freeze_modules.py
@echo "The Makefile was updated, you may need to re-run make."

# BEGIN: freezing modules

Python/frozen_modules/importlib__bootstrap.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/importlib/_bootstrap.py
@echo "Freezing Python/frozen_modules/importlib__bootstrap.h from Lib/importlib/_bootstrap.py"
@$(srcdir)/Programs/_freeze_module importlib._bootstrap \
$(srcdir)/Lib/importlib/_bootstrap.py \
$(srcdir)/Python/frozen_modules/importlib__bootstrap.h

Python/frozen_modules/importlib__bootstrap_external.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/importlib/_bootstrap_external.py
@echo "Freezing Python/frozen_modules/importlib__bootstrap_external.h from Lib/importlib/_bootstrap_external.py"
@$(srcdir)/Programs/_freeze_module importlib._bootstrap_external \
$(srcdir)/Lib/importlib/_bootstrap_external.py \
$(srcdir)/Python/frozen_modules/importlib__bootstrap_external.h

Python/frozen_modules/zipimport.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/zipimport.py
@echo "Freezing Python/frozen_modules/zipimport.h from Lib/zipimport.py"
@$(srcdir)/Programs/_freeze_module zipimport \
$(srcdir)/Lib/zipimport.py \
$(srcdir)/Python/frozen_modules/zipimport.h

Python/frozen_modules/abc.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/abc.py
@echo "Freezing Python/frozen_modules/abc.h from Lib/abc.py"
@$(srcdir)/Programs/_freeze_module abc \
$(srcdir)/Lib/abc.py \
$(srcdir)/Python/frozen_modules/abc.h

Python/frozen_modules/codecs.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/codecs.py
@echo "Freezing Python/frozen_modules/codecs.h from Lib/codecs.py"
@$(srcdir)/Programs/_freeze_module codecs \
$(srcdir)/Lib/codecs.py \
$(srcdir)/Python/frozen_modules/codecs.h

Python/frozen_modules/io.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/io.py
@echo "Freezing Python/frozen_modules/io.h from Lib/io.py"
@$(srcdir)/Programs/_freeze_module io \
$(srcdir)/Lib/io.py \
$(srcdir)/Python/frozen_modules/io.h

Python/frozen_modules/_collections_abc.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/_collections_abc.py
@echo "Freezing Python/frozen_modules/_collections_abc.h from Lib/_collections_abc.py"
@$(srcdir)/Programs/_freeze_module _collections_abc \
$(srcdir)/Lib/_collections_abc.py \
$(srcdir)/Python/frozen_modules/_collections_abc.h

Python/frozen_modules/_sitebuiltins.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/_sitebuiltins.py
@echo "Freezing Python/frozen_modules/_sitebuiltins.h from Lib/_sitebuiltins.py"
@$(srcdir)/Programs/_freeze_module _sitebuiltins \
$(srcdir)/Lib/_sitebuiltins.py \
$(srcdir)/Python/frozen_modules/_sitebuiltins.h

Python/frozen_modules/genericpath.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/genericpath.py
@echo "Freezing Python/frozen_modules/genericpath.h from Lib/genericpath.py"
@$(srcdir)/Programs/_freeze_module genericpath \
$(srcdir)/Lib/genericpath.py \
$(srcdir)/Python/frozen_modules/genericpath.h

Python/frozen_modules/posixpath.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/posixpath.py
@echo "Freezing Python/frozen_modules/posixpath.h from Lib/posixpath.py"
@$(srcdir)/Programs/_freeze_module posixpath \
$(srcdir)/Lib/posixpath.py \
$(srcdir)/Python/frozen_modules/posixpath.h

Python/frozen_modules/os.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/os.py
@echo "Freezing Python/frozen_modules/os.h from Lib/os.py"
@$(srcdir)/Programs/_freeze_module os \
$(srcdir)/Lib/os.py \
$(srcdir)/Python/frozen_modules/os.h

Python/frozen_modules/site.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/site.py
@echo "Freezing Python/frozen_modules/site.h from Lib/site.py"
@$(srcdir)/Programs/_freeze_module site \
$(srcdir)/Lib/site.py \
$(srcdir)/Python/frozen_modules/site.h

Python/frozen_modules/stat.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Lib/stat.py
@echo "Freezing Python/frozen_modules/stat.h from Lib/stat.py"
@$(srcdir)/Programs/_freeze_module stat \
$(srcdir)/Lib/stat.py \
$(srcdir)/Python/frozen_modules/stat.h

Python/frozen_modules/hello.h: $(srcdir)/Programs/_freeze_module $(srcdir)/Tools/freeze/flag.py
@echo "Freezing Python/frozen_modules/hello.h from Tools/freeze/flag.py"
@$(srcdir)/Programs/_freeze_module hello \
$(srcdir)/Tools/freeze/flag.py \
$(srcdir)/Python/frozen_modules/hello.h

# END: freezing modules

# We keep this renamed target around for folks with muscle memory.
.PHONY: regen-importlib
regen-importlib: regen-frozen
Expand Down Expand Up @@ -1330,25 +1242,7 @@ regen-opcode-targets:
Python/ceval.o: $(srcdir)/Python/opcode_targets.h $(srcdir)/Python/ceval_gil.h \
$(srcdir)/Python/condvar.h

# FROZEN_FILES is auto-generated by Tools/scripts/freeze_modules.py.
FROZEN_FILES = \
$(srcdir)/Python/frozen_modules/importlib__bootstrap.h \
$(srcdir)/Python/frozen_modules/importlib__bootstrap_external.h \
$(srcdir)/Python/frozen_modules/zipimport.h \
$(srcdir)/Python/frozen_modules/abc.h \
$(srcdir)/Python/frozen_modules/codecs.h \
$(srcdir)/Python/frozen_modules/io.h \
$(srcdir)/Python/frozen_modules/_collections_abc.h \
$(srcdir)/Python/frozen_modules/_sitebuiltins.h \
$(srcdir)/Python/frozen_modules/genericpath.h \
$(srcdir)/Python/frozen_modules/posixpath.h \
$(srcdir)/Python/frozen_modules/os.h \
$(srcdir)/Python/frozen_modules/site.h \
$(srcdir)/Python/frozen_modules/stat.h \
$(srcdir)/Python/frozen_modules/hello.h
# End FROZEN_FILES

Python/frozen.o: $(FROZEN_FILES)
Python/frozen.o: $(FROZEN_FILES_OUT)

# Generate DTrace probe macros, then rename them (PYTHON_ -> PyDTrace_) to
# follow our naming conventions. dtrace(1) uses the output filename to generate
Expand Down
1 change: 1 addition & 0 deletions Modules/getpath.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1495,6 +1495,7 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig)

if (pathconfig->stdlib_dir == NULL) {
if (calculate->prefix_found) {
/* This must be done *before* calculate_set_prefix() is called. */
pathconfig->stdlib_dir = _PyMem_RawWcsdup(calculate->prefix);
if (pathconfig->stdlib_dir == NULL) {
return _PyStatus_NO_MEMORY();
Expand Down
34 changes: 32 additions & 2 deletions PCbuild/_freeze_module.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,12 +272,17 @@
</None>
<None Include="..\Lib\genericpath.py">
<ModName>genericpath</ModName>
<IntFile>$(IntDir)genericpat.g.h</IntFile>
<IntFile>$(IntDir)genericpath.g.h</IntFile>
<OutFile>$(PySourcePath)Python\frozen_modules\genericpath.h</OutFile>
</None>
<None Include="..\Lib\ntpath.py">
<ModName>ntpath</ModName>
<IntFile>$(IntDir)ntpath.g.h</IntFile>
<OutFile>$(PySourcePath)Python\frozen_modules\ntpath.h</OutFile>
</None>
<None Include="..\Lib\posixpath.py">
<ModName>posixpath</ModName>
<IntFile>$(IntDir)posixpat.g.h</IntFile>
<IntFile>$(IntDir)posixpath.g.h</IntFile>
<OutFile>$(PySourcePath)Python\frozen_modules\posixpath.h</OutFile>
</None>
<None Include="..\Lib\os.py">
Expand All @@ -295,6 +300,31 @@
<IntFile>$(IntDir)stat.g.h</IntFile>
<OutFile>$(PySourcePath)Python\frozen_modules\stat.h</OutFile>
</None>
<None Include="..\Lib\__hello__.py">
<ModName>__hello__</ModName>
<IntFile>$(IntDir)__hello__.g.h</IntFile>
<OutFile>$(PySourcePath)Python\frozen_modules\__hello__.h</OutFile>
</None>
<None Include="..\Lib\__phello__\__init__.py">
<ModName>__phello__</ModName>
<IntFile>$(IntDir)__phello__.g.h</IntFile>
<OutFile>$(PySourcePath)Python\frozen_modules\__phello__.h</OutFile>
</None>
<None Include="..\Lib\__phello__\ham\__init__.py">
<ModName>__phello__.ham</ModName>
<IntFile>$(IntDir)__phello__.ham.g.h</IntFile>
<OutFile>$(PySourcePath)Python\frozen_modules\__phello__.ham.h</OutFile>
</None>
<None Include="..\Lib\__phello__\ham\eggs.py">
<ModName>__phello__.ham.eggs</ModName>
<IntFile>$(IntDir)__phello__.ham.eggs.g.h</IntFile>
<OutFile>$(PySourcePath)Python\frozen_modules\__phello__.ham.eggs.h</OutFile>
</None>
<None Include="..\Lib\__phello__\spam.py">
<ModName>__phello__.spam</ModName>
<IntFile>$(IntDir)__phello__.spam.g.h</IntFile>
<OutFile>$(PySourcePath)Python\frozen_modules\__phello__.spam.h</OutFile>
</None>
<None Include="..\Tools\freeze\flag.py">
<ModName>frozen_only</ModName>
<IntFile>$(IntDir)frozen_only.g.h</IntFile>
Expand Down
18 changes: 18 additions & 0 deletions PCbuild/_freeze_module.vcxproj.filters
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@
<None Include="..\Lib\genericpath.py">
<Filter>Python Files</Filter>
</None>
<None Include="..\Lib\ntpath.py">
<Filter>Python Files</Filter>
</None>
<None Include="..\Lib\posixpath.py">
<Filter>Python Files</Filter>
</None>
Expand All @@ -55,6 +58,21 @@
<None Include="..\Lib\stat.py">
<Filter>Python Files</Filter>
</None>
<None Include="..\Lib\__hello__.py">
<Filter>Python Files</Filter>
</None>
<None Include="..\Lib\__phello__\__init__.py">
<Filter>Python Files</Filter>
</None>
<None Include="..\Lib\__phello__\ham\__init__.py">
<Filter>Python Files</Filter>
</None>
<None Include="..\Lib\__phello__\ham\eggs.py">
<Filter>Python Files</Filter>
</None>
<None Include="..\Lib\__phello__\spam.py">
<Filter>Python Files</Filter>
</None>
<None Include="..\Tools\freeze\flag.py">
<Filter>Python Files</Filter>
</None>
Expand Down
19 changes: 14 additions & 5 deletions Python/bootstrap_frozen.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,28 @@
which is used (only) to run Tools/scripts/deepfreeze.py. */

#include "Python.h"
#include "pycore_import.h"

#include "frozen_modules/importlib__bootstrap.h"
#include "frozen_modules/importlib__bootstrap_external.h"
#include "frozen_modules/importlib._bootstrap.h"
#include "frozen_modules/importlib._bootstrap_external.h"
#include "frozen_modules/zipimport.h"

static const struct _frozen _PyImport_FrozenModules[] = {
/* importlib */
/* import system */
{"_frozen_importlib", _Py_M__importlib__bootstrap,
(int)sizeof(_Py_M__importlib__bootstrap)},
(int)sizeof(_Py_M__importlib__bootstrap)},
{"_frozen_importlib_external", _Py_M__importlib__bootstrap_external,
(int)sizeof(_Py_M__importlib__bootstrap_external)},
(int)sizeof(_Py_M__importlib__bootstrap_external)},
{"zipimport", _Py_M__zipimport, (int)sizeof(_Py_M__zipimport)},

{0, 0, 0} /* sentinel */
};

static const struct _module_alias aliases[] = {
{"_frozen_importlib", "importlib._bootstrap"},
{"_frozen_importlib_external", "importlib._bootstrap_external"},
{0, 0} /* aliases sentinel */
};
const struct _module_alias *_PyImport_FrozenAliases = aliases;

const struct _frozen *PyImport_FrozenModules = _PyImport_FrozenModules;
Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.