shlex include empty strings

Question

sample = ",,"
values = shlex.shlex(sample, posix=True)
values.quotes = '"'
values.whitespace = ','
values.whitespace_split = True

received_output = list(values)

In the above code sample I would like to have ["", "", ""] as the value of received_output, but received_output is simply an empty list []. There does not seem to be any information regarding how to receive this expected behavior.

This works fine with sample.split(','), but I would prefer using shlex since I have complex sentences with tokens which should not be split up if part of a group (such as latitude, longitude in the following example).

Another example:

sample = '9267,BELMONT,KEELER,,62.4,35.2,10/01/2012,Weekday,"(41.93897000, -87.73212000)"'

expected_output = ['9267', 'BELMONT', 'KEELER', '', '62.4', '35.2', '10/01/2012', 'Weekday', '(41.93897000, -87.73212000)']
retrieved_output = ['9267', 'BELMONT', 'KEELER', '62.4', '35.2', '10/01/2012', 'Weekday', '(41.93897000, -87.73212000)']

Answer 1

The shlex docs state:

• It’s not possible to parse empty strings, even if quoted.

If you want to include empty strings in your output, the shlex library is the wrong tool for the job.

As @PadraicCunningham points out in a comment, the csv (Comma Separated Values) library should work fine for this:

>>> list(csv.reader(['9267,BELMONT,KEELER,,62.4,35.2,10/01/2012,Weekday,"(41.93897000, -87.73212000)"']))[0]
['9267', 'BELMONT', 'KEELER', '', '62.4', '35.2', '10/01/2012', 'Weekday', '(41.93897000, -87.73212000)']
>>> list(csv.reader([',,']))[0]
['', '', '']

Answer 2

So I initially upvoted the other answer, but it turns out it's wrong. The quote from the documentation is from the section on non-Posix behavior, but by default Posix behavior is used. With Posix behavior you can get an empty string like this:

shlex.split('x ""')
# ['x', '']

Note that quotes don't separate words in Posix mode, so this happens:

shlex.split('x""')
# ['x']

This is a little confusing, but you can still get empty strings with shlex.

In your particular case, since you seem to be parsing CSV-like input anyway, using the CSV module makes sense.