argparse: flatten the result of action='append'

Question

I'd like to make a script that supports an argument list of the form

./myscript --env ONE=1,TWO=2 --env THREE=3

Here's my attempt:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument(
    '--env',
    type=lambda s: s.split(','),
    action='append',
)
options = parser.parse_args()
print options.env

$ ./myscript --env ONE=1,TWO=2 --env THREE=3
[['ONE=1', 'TWO=2'], ['THREE=3']]

Sure I can fix this in postprocessing:

options.env = [x for y in options.env for x in y]

but I'm wondering if there's some way to get the flattened list directly from argparse, so that I don't have to maintain a list of "things I need to flatten afterwards" in my head as I'm adding new options to the program.

The same question applies if I were to use nargs='*' instead of type=lambda....

import argparse
parser = argparse.ArgumentParser()
parser.add_argument(
    '--env',
    nargs='+',
    action='append',
)
options = parser.parse_args()
print options.env

$ ./myscript --env ONE=1 TWO=2 --env THREE=3
[['ONE=1', 'TWO=2'], ['THREE=3']]

Answer 1

Edit: Since Python 3.8, the "extend" is available directly in stdlib. If you only have to support 3.8+ then defining it yourself is no longer required. Usage of stdlib "extend" action is exactly the same way as this answer originally described:

>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> _ = parser.add_argument('--env', nargs='+', action='extend')
>>> parser.parse_args(["--env", "ONE", "TWO", "--env", "THREE"])
Namespace(env=['ONE', 'TWO', 'THREE'])

---

Unfortunately, there isn't an extend action provided in ArgumentParser by default. But it's not too hard to register one:

import argparse

class ExtendAction(argparse.Action):

    def __call__(self, parser, namespace, values, option_string=None):
        items = getattr(namespace, self.dest) or []
        items.extend(values)
        setattr(namespace, self.dest, items)


parser = argparse.ArgumentParser()
parser.register('action', 'extend', ExtendAction)
parser.add_argument('--env', nargs='+', action='extend')

args = parser.parse_args()
print(args)

Demo:

$ python /tmp/args.py --env one two --env three
Namespace(env=['one', 'two', 'three'])

The lambda you have in your example is somewhat outside the intended use-case of the type kwarg. So, I would recommend instead to split on whitespace, because it will be a pain to correctly handle the case where , is actually in the data. If you split on space, you get this functionality for free:

$ python /tmp/args.py --env one "hello world" two --env three
Namespace(env=['one', 'hello world', 'two', 'three'])

Answer 2

An extend action class has been asked for (http://bugs.python.org/issue23378), but since it's easy to add your own I don't think the feature will ever by added.

A common Python idiom for flattening a list uses chain:

In [36]: p=[['x'], ['y']]
In [37]: from itertools import chain
In [38]: chain(*p)
Out[38]: <itertools.chain at 0xb17afb2c>
In [39]: list(chain(*p))
Out[39]: ['x', 'y']

Your list comprehension is the equivalent:

In [40]: [x for y in p for x in y]
Out[40]: ['x', 'y']

In http://bugs.python.org/issue16399#msg277919 I suggest another possiblity - a custom default value for the append argument.

class MyList(list):
    def append(self,arg):
        if isinstance(arg,list):
            self.extend(arg)
        else:
            super(MyList, self).append(arg)

 parser = argparse.ArgumentParser()
 a = parser.add_argument('-f', action='append', nargs='*',default=MyList([]))
 args = parser.parse_args('-f 1 2 3 -f 4 5'.split())

which produces

Namespace(f=['1', '2', '3', '4', '5'])

You'll have to use your own judgment as to what's appropriate in production code.

Answer 3

Here is the one that solves but not directly from argparse

#!/usr/bin/python
import argparse
parser = argparse.ArgumentParser()
parser.add_argument(
    '--env',
    nargs='+',
    action='append',
)
options = parser.parse_args()

options.env = [item for args in options.env for item in args]

print options.env

Output is this:

$ ./argtest.py --env ONE=1 TWO=2 --env THREE=3

['ONE=1', 'TWO=2', 'THREE=3']