https://docs.python.org/3/library/shlex.html#improved-compatibility-with-shells

The code sample here does not match the output - the first line is labelled 'New' and the second line 'Old'.

Thank you. The patch fixes it and makes the example under doctest.

Sorry for not being more clear in the original report - the error is in the code, not in the output. The old behavior is punct=False, the new is punct=True.

Yes, you are right. Here is a patch.

Second patch looks good, thanks. Do you also want to doctest that?

I did not add the doctest directive on purpose. In fact, if in the future we start running the doctests with buildbot (see issue27200), the tests related to this issue will pass only for Python >= 3.6. If we _currently_ want the doctests to pass for every Python version, I see only two solutions:

1) we do not have to doctest the example, as I did in issue29133_2nd.patch
2) we have to apply two patches, one with doctests (for the doc to be tested with Python >= 3.6), and another one without doctests (for the doc to be tested with Python < 3.6).

I think the best solution is to add an option to the doctest directive, that allows us to specify for which Python version the tests have to be run:

https://github.com/sphinx-doc/sphinx/issues/3303

I would like the commiters to give a feedback about this, because if they think  this solution is doable, then I will spend the time to work on the Sphinx issue.

I'd probably write it without the for loop:

    text = "a && b; c && d || e; f >'abc'; (def \"ghi\")"

    result = shlex.shlex(text)
    print(f"Old behavior: {list(result)}")

    result = shlex.shlex(text, punctuation_chars=True)
    print(f"New behavior: {list(result)}")

Or just:

    >>> import shlex
    >>> text = "a && b; c && d || e; f >'abc'; (def \"ghi\")"
    >>> list(shlex.shlex(text))
    ['a', '&', '&', 'b', ';', 'c', '&', '&', 'd', '|', '|', 'e', ';', 'f', '>', "'abc'", ';', '(', 'def', '"ghi"', ')']
    >>> list(shlex.shlex(text, punctuation_chars=True))
    ['a', '&&', 'b', ';', 'c', '&&', 'd', '||', 'e', ';', 'f', '>', "'abc'", ';', '(', 'def', '"ghi"', ')']

(Adding Vinay to nosy list to get his feedback since he wrote the original example.)

I'm fine with removing the loop, as there are only ever going to be the two cases - so Berker's suggestion would seem to cover both doctest and unittest cases.

Here is a 3rd patch with the Berker's suggestion. I just limited the output to 79 characters per line, and made use of the +NORMALIZE_WHITESPACE option  (to normalize the newline inside the output).

New changeset af5b34b2d169 by Vinay Sajip in branch '3.6':
Fixes #29133: clarified shlex documentation.
https://hg.python.org/cpython/rev/af5b34b2d169

New changeset e3d820c0c884 by Vinay Sajip in branch 'default':
Closes #29133: merged update from 3.6.
https://hg.python.org/cpython/rev/e3d820c0c884