Skip to content

bpo-40958: Avoid buffer overflow in the parser when indexing the current line#20875

Merged
pablogsal merged 4 commits intopython:masterfrom
pablogsal:bpo-40958
Jun 16, 2020
Merged

bpo-40958: Avoid buffer overflow in the parser when indexing the current line#20875
pablogsal merged 4 commits intopython:masterfrom
pablogsal:bpo-40958

Conversation

@pablogsal
Copy link
Member

@pablogsal pablogsal commented Jun 15, 2020 •
edited
Loading

After thinking a bit more about the whole problem I found that we were processing the offsets incorrectly if the input is raw (no need to transform the offset to a character offset). Once we process the column offsets correctly, is quite straightforward to fix the incorrect access.

As an example of why we were processing the offset incorrectly previously, consider this code (current master without this PR):

❯ ./python
Python 3.9.0b1+ (heads/3.9:299d3d1c52, Jun  8 2020, 22:11:46)
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> Python = "Ṕýţĥòñ" + 34 $ lelummsmdsads +
  File "<stdin>", line 1
    Python = "Ṕýţĥòñ" + 34 $ lelummsmdsads +
                                  ^
SyntaxError: invalid syntax

while the old parser points correctly to the $ token:

❯ ./python -Xoldparser
Python 3.9.0b1+ (heads/3.9:299d3d1c52, Jun  8 2020, 22:11:46)
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> Python = "Ṕýţĥòñ" + 34 $ lelummsmdsads +
  File "<stdin>", line 1
    Python = "Ṕýţĥòñ" + 34 $ lelummsmdsads +
                           ^
SyntaxError: invalid syntax

https://bugs.python.org/issue40958

@pablogsal pablogsal requested review from lysnikolaou and tiran June 15, 2020 00:30
@pablogsal pablogsal self-assigned this Jun 15, 2020
tiran
tiran requested changes Jun 15, 2020
View reviewed changes
@bedevere-bot
Copy link

bedevere-bot commented Jun 15, 2020

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

@pablogsal
Copy link
Member Author

pablogsal commented Jun 15, 2020

I have made the requested changes; please review again

@bedevere-bot
Copy link

bedevere-bot commented Jun 15, 2020

Thanks for making the requested changes!

@tiran: please review the changes made to this pull request.

@bedevere-bot bedevere-bot requested a review from tiran June 15, 2020 10:12
lysnikolaou
lysnikolaou reviewed Jun 15, 2020
View reviewed changes
Copy link
Member

@lysnikolaou lysnikolaou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 🎉

lysnikolaou
lysnikolaou approved these changes Jun 15, 2020
View reviewed changes
Copy link
Member

@lysnikolaou lysnikolaou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks thanks!

@pablogsal
Copy link
Member Author

pablogsal commented Jun 16, 2020

@tiran If you have some time today, could you confirm that this solves the ASAN/USAN problem for you? I have checked on my side that this is the case but it would be nice to have independent confirmation. I plan to land this today and do the backport to avoid potential merge conflicts.

@tiran
Copy link
Member

tiran commented Jun 16, 2020

@tiran If you have some time today, could you confirm that this solves the ASAN/USAN problem for you? I have checked on my side that this is the case but it would be nice to have independent confirmation. I plan to land this today and do the backport to avoid potential merge conflicts.

After a rebase to latest master the PR is passing asan/ubsan tests on my F32 machine.

@pablogsal pablogsal merged commit 51c5896 into python:master Jun 16, 2020
@miss-islington
Copy link
Contributor

miss-islington commented Jun 16, 2020

Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

@pablogsal pablogsal deleted the bpo-40958 branch June 16, 2020 15:49
@bedevere-bot
Copy link

bedevere-bot commented Jun 16, 2020

GH-20919 is a backport of this pull request to the 3.9 branch.

pablogsal added a commit to miss-islington/cpython that referenced this pull request Jun 16, 2020
@pablogsal
bpo-40958: Avoid buffer overflow in the parser when indexing the curr…
218c707 
…ent line (pythonGH-20875)

(cherry picked from commit 51c5896)

Co-authored-by: Pablo Galindo <[email protected]>
pablogsal added a commit that referenced this pull request Jun 16, 2020
@miss-islington @pablogsal
bpo-40958: Avoid buffer overflow in the parser when indexing the curr…
7795ae8 
…ent line (GH-20875) (GH-20919)

(cherry picked from commit 51c5896)

Co-authored-by: Pablo Galindo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tiran tiran tiran approved these changes

@lysnikolaou lysnikolaou lysnikolaou approved these changes

Assignees

@pablogsal pablogsal

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@pablogsal @bedevere-bot @tiran @miss-islington @lysnikolaou @the-knights-who-say-ni