shlex preserving double quotes?

Question

I'm using Popen with shlex for a yum command with the --exclude flag to pass a list of packages to be excluded. For some reason it seems shlex is not preserving the double quotes. Any pointers how do i go about this ?

>>> import shlex
>>> x = '/usr/bin/yum update --exclude=\"foo bar baz*\"'   
>>> print shlex.split(x)
['/usr/bin/yum', 'update', '--exclude=foo bar baz*']

With the POSIX mode off, the quotes seem misplaced.

>>> print shlex.split(x,posix=False)
['/usr/bin/yum', 'update', '--exclude="foo', 'bar', 'baz*"']

Answer 1

First, your parameters will already arrive correct in the target program's argv/argc via normal shlex.split(x) - as shown here :

>>> x = '/usr/bin/yum update --exclude=\"foo bar baz*\"' 
>>> l = shlex.split(x); l
['/usr/bin/yum', 'update', '--exclude=foo bar baz*']
>>> p = subprocess.Popen(['python', '-c', "import sys;print(sys.argv)"] + l, stdout=subprocess.PIPE)
>>> p.stdout.read()
"['-c', '/usr/bin/yum', 'update', '--exclude=foo bar baz*']\r\n"

: The quotes on the shell are there for keeping the parameter string together for correct splitting in the target app; and with split arguments, the split is already done.

---

Second, you can use shell=True on Popen and you can pass the command string unsplit:

p = subprocess.Popen(x, shell=True, ...)

: This way the string x is directly interpreted by the shell - e.g. bash. But an extra shell process is involved and consumes resources and run time

---

Third, if you really want to get the quotes your way into the target program via shlex.split and direct program call, and when you have control over the input string, then you can the write the input string like this (Extra quoting the same as in shell(-mode)):

>>> x = '/usr/bin/yum update --exclude="\\"foo bar baz*\\""' 
>>> l = shlex.split(x); l
['/usr/bin/yum', 'update', '--exclude="foo bar baz*"']
>>> p = subprocess.Popen(['python', '-c', "import sys;print(sys.argv)"] + l, stdout=subprocess.PIPE)
>>> print(p.stdout.read())
['-c', '/usr/bin/yum', 'update', '--exclude="foo bar baz*"']

(note: one backslash in the input string is consume by python syntax unless your write it in r-syntax: x = r'/usr/bin/yum update --exclude="\"foo bar baz*\\""'

---

Fourth, if you want get your given input string with the quotes into the target programm, then you need to employ a custom shell syntax - e.g. via a REGEX:

>>> x = '/usr/bin/yum update --exclude=\"foo bar baz*\"' 
>>> l = re.findall(r'(?:[^\s,"]|"(?:\\.|[^"])*")+', x); l
['/usr/bin/yum', 'update', '--exclude="foo bar baz*"']
>>> p = subprocess.Popen(['python', '-c', "import sys;print(sys.argv)"] + l, stdout=subprocess.PIPE)
>>> print(p.stdout.read())
['-c', '/usr/bin/yum', 'update', '--exclude="foo bar baz*"']