Optional arguments across all subparsers

Question

I'm currently testing an argparse usage, but it's not working as expected. I have a couple of subparsers and optional arguments, being called the following way:

python3 myprogram.py positional argument --optional something

# Outcome
Namespace(optional='something')

The program works as expected if the optional is the last, but if it is in any other order, it is discarded.

python3 myprogram.py positional --optional argument
python3 myprogram.py --optional positional argument

# Outcome
Namespace(optional=None)

By looking at the argparse documentation I wasn't able to find a way to make the optional argument global.

I'm creating the the positional arguments for each positional in a for loop, which doesn't seem to be the best way. Because otherwise, it would add the optional arguments only to the last subparser.

import argparse

class Parsing(object):

    def __init__(self):

        parser = argparse.ArgumentParser(prog='python3 myprogram.py',
            formatter_class=argparse.RawDescriptionHelpFormatter,
            description='some description')

        self.subparser = parser.add_subparsers(title='Positional', help='help description')

        for sub in self.Generate(): # Method with a bunch of subparsers
            self.Subparser(sub)

    def Subparser(self, parsers):

        for each in sorted(parsers):
            positional = subparser.add_parser(each)
            self.Optional(positional) # Method with some optional arguments for each of the second subparsers

        self.Optional(parser) # Adding the optional arguments to the first subparser

    def Optional(self, parser):

        # ... Optional arguments

    def Generate(self):

        # ... Subparsers

I might be missing some code in the example above, tried to simplify the best I could and hope it to be perceptible.

Question: Is there a way to make the optional arguments across all subparsers?

Answer 1

Your description and code is hard to follow, but I've concluded that your problem lies with how defaults are handled when the main and subparsers share an argument dest.

I condensed your code a bit so I could make a test run:

import argparse
class Parsing(object):
    def __init__(self):
        self.parser = argparse.ArgumentParser(prog='prog',
            description='some description')
        self.subparser = self.parser.add_subparsers(dest='cmd', title='Cmds', help='help description')
        self.make_subparsers(['cmd1','cmd2'])

    def make_subparsers(self, parsers):
        for each in parsers:
            subp = self.subparser.add_parser(each)
            self.optional(subp, default='sub') 
        self.optional(self.parser, default='main') 

    def optional(self, parser, default=None):
        parser.add_argument('--foo', default=default)

args = Parsing().parser.parse_args()
print(args)

I get for 2 runs

1315:~/mypy$ python3.5 stack41431025.py cmd1 --foo 1
Namespace(cmd='cmd1', foo='1')

1316:~/mypy$ python3.5 stack41431025.py --foo 1 cmd1
Namespace(cmd='cmd1', foo='sub')

In the first, foo is set by the strings parsed by the cmd1 subparser.

In the second, foo gets the default value set by the subparser. The main parser parsed --foo, but its value was over written by the subparser.

There has been some discussion of this in bug/issues. http://bugs.python.org/issue9351 changed handling so that the subparser default has priority over main parser values. I think there are problems with that patch, but it's been in effect for a couple of years.

You retain more control if they are given different dest.

def make_subparsers(self, parsers):
    for each in parsers:
        subp = self.subparser.add_parser(each)
        self.optional(subp, default='sub') 
    self.optional(self.parser, default='main', dest='main_foo') 

def optional(self, parser, default=None, dest=None):
    parser.add_argument('--foo', default=default, dest=dest)

1325:~/mypy$ python3.5 stack41431025.py --foo 1 cmd1
Namespace(cmd='cmd1', foo='sub', main_foo='1')
1325:~/mypy$ python3.5 stack41431025.py cmd1
Namespace(cmd='cmd1', foo='sub', main_foo='main')
1325:~/mypy$ python3.5 stack41431025.py --foo 1 cmd1 --foo 2
Namespace(cmd='cmd1', foo='2', main_foo='1')

====================

(earlier answer)

I'll try to sketch the possible combinations of arguments

parser = argparse.ArgumentParser()
parser.add_argument('mainpos', help='positional for main')
parser.add_argument('--mainopt', help='optional defined for main')
sp = parser.add_subparser(dest='cmd')
p1 = sp.add_parser('cmd1')
p1.add_argument('subpos', help='postional for sub')
p1.add_argument('--subopt', help='optional defined for sub')

A composite usage would look like:

python prog.py foo [--mainopt bar] cmd1 sfoo [--subopt baz]

The respective positionals have to be given in the right order. The subparser cmd is effectively a positional for the main.

The optional defined for main has to occur before the subparser name. The optional defined for the subparser has to occur after. They could have the same flag or dest, but they have to be defined separately. And if they have the same dest, there could be a conflict over values, especially defaults.

parser.parse_args() starts matching the input strings with its arguments. If it sees --mainopt is parses that optional argument. Otherwise it expects two postionals. The 2nd has to be one of the subparser names.

Once it gets a subparser name it passes the remaining strings to that subparser. The subparser handles the rest, and puts the values in the main namespace. And the first thing the subparser does is set its defaults. Whether that action overwrites values set by the main parser or not depends on just how the namespace is passed between the two.

================

Parsing is driven by the order of arguments in the command line. It tries to allow flagged arguments in any order. But once parsing is passed to the subparser, the main parser does not get another go at parsing. It just does a few clean up tasks.

But if I use parse_known_args, I can collect the strings that neither parser handled, and take another stab a parsing them.

parser1 = argparse.ArgumentParser()
parser1.add_argument('--foo')
sp = parser1.add_subparsers(dest='cmd')
sp1 = sp.add_parser('cmd1')
args, extra = parser1.parse_known_args()

parser2 = argparse.ArgumentParser()
parser2.add_argument('--foo')
if extra:
    args = parser2.parse_args(extra)
print(args)

runs

1815:~/mypy$ python stack41431025.py --foo 1 cmd1
Namespace(cmd='cmd1', foo='1')

1815:~/mypy$ python stack41431025.py cmd1 --foo 2
Namespace(foo='2')

1815:~/mypy$ python stack41431025.py --foo 1 cmd1 --foo 3
Namespace(foo='3')

I haven't tested this idea in anything more complex, so there might be some interactions that I haven't thought of. But this is the closest I can come to a flagged argument that can occur anywhere, and not be subject to conflicting default problems.