Split multi-line string with shlex and keep quote characters

Question

How do I split a string with Python's shlex while preserving the quote characters that shlex splits on?

Sample Input:

Two Words
"A Multi-line
 comment."

Desired Output:

['Two', 'Words', '"A Multi-line\ncomment."']

Note the double quotes wrapping the multi-line string. I read through the shlex documentation, but I don't see an obvious option. Does this require a regular expression solution?

Answer 1

I'm not sure why you're trying to use shlex for this. The whole point is to split into the same arguments the shell would. As far as the shell is concerned, those quotes are not part of the argument. So, this is probably the wrong thing to do…

But if you want to do it, you can access the lower levels of the shlex parser, which makes this trivial. For example:

>>> data = '''Two Words
"A Multi-line
 comment."'''
>>> sh = shlex.shlex(data)
>>> sh.get_token()
'Two'
>>> sh.get_token()
'Words'
>>> sh.get_token()
'"A Multi-line\n comment."'

    >>> sh.get_token()     ''

So, if you want to get this as a list, you can do this one-liner:

>>> list(iter(shlex.shlex(data).get_token, ''))

I believe this requires Python 2.3+, but since you linked to the docs from 3.4 I doubt that's a problem. Anyway, I verified that it works in both 2.7 and 3.3.

Answer 2

>>> print(s)
Two Words
"A Multi-line
 comment."
>>> shlex.split(s)
['Two', 'Words', 'A Multi-line\n comment.']
>>> shlex.split(s, posix=False)
['Two', 'Words', '"A Multi-line\n comment."']
>>> 

Changed in version 2.6: Added the posix parameter.