Fix benchmarks by replacing rust-cpython with pyo3#5163
Fix benchmarks by replacing rust-cpython with pyo3#5163youknowone merged 2 commits intoRustPython:mainfrom
Conversation
The benchmarks have been broken since Python 3.10 deprecated the API they were using to parse and execute CPython baselines. Since then rust-cpython has been deprecated in favor of pyo3. - Replace `cpython` and `python3-sys` with `pyo3`. - Add `html_report` feature to `criterion`, it will be required in a future release. - Remove `anyhow`. It was unused and cargo cleaned it up automatically.
Update the actual benchmark harnesses. Because the internal APIs previously used are no longer available, I opted to use `compile` and `exec` from within the CPython context to compile and execute code. There's probably more overhead to that than the internal API had, but that overhead should be consistent per benchmark. If anyone cares about hyperoptimizing benchmarks then they can optimize the harness as well.
I have a minor question: How do we know that this won't inadvertently give RustPython an advantage? As far as I know, the API we used (from before CPython 3.10) ensured that compilation was kept separate from execution. |
|
Also, if I recall correctly, the RustPython benchmarks will eat up a lot of memory because RustPython presently has no garbage collection more advanced than simple reference-counting and needs circular references in order to set up simple things like |
|
When I originally tried to run the benchmarks they just failed to build because of features deprecated in Python 3.10, so my understanding was that they just haven't worked since the project moved past that point. My objective with this PR was to get something working so that I could benchmark some changes I was making. From my cursory investigation, I couldn't find that pyo3 exposed a similar API to what was there previously. I think the difference between calling the CPython I'm bummed about garbage collection too, but I don't see that it should block improving the benchmarking code. |
|
The reason I warned about GC is so that you're not surprised if the benchmarks crash due to your system not having enough RAM. (From what I recall, RustPython chews through about 16 GB of RAM.) |
|
Thanks for the warning! That is probably why I crashed my computer a few times testing this PR lol |
youknowone
left a comment
youknowone
left a comment
There was a problem hiding this comment.
Thank you! it was broken for years and finally going to be fixed
3 participants
The benchmarks seem to have been broken since Python 3.10 deprecated the API
they were using to parse and execute CPython baselines. Since then
rust-cpython has been deprecated in favor of pyo3.
cpythonandpython3-sysdependencies withpyo3.html_reportfeature tocriterion, it will be required in afuture release.
anyhow. It was unused and cargo cleaned it up automatically.compileandexecfrom within the CPython context tocompile and execute benchmarks. There's probably more overhead to that than
the internal API had, but that overhead should be consistent per
benchmark.
If anyone cares about hyperoptimizing benchmarks then they can optimize
the harness as well.
Fixes #3498