How to create subparser with argparse from existing program in Python 3?

Question

Original post:

If one has an executable mini_program.py that uses argparse with the following structure:

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('-X', '--attribute_matrix', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
    parser.add_argument('-y', '--target_vector', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
    opts = parser.parse_args()

if __name__ == "__main__":
    main()

How can one create a controller program parent_program.py that uses argparse (I think with subparser?) to have a similar usage to below:

python parent_program.py --help

blah-blah list of programs that can be used

then using the subprogram:

python parent_program.py mini_program --help

-X description
-y description
etc...

How could all of the parameters propagate up from mini_program.py to the parent_program.py?

EDIT (More specific with error message):

The program

import argparse
def main():
    parser = argparse.ArgumentParser()
    # Subprograms
    subprograms = parser.add_subparsers(title="subprograms")
    # ============
    # mini-program
    # ============
    parser_miniprogram = subprograms.add_parser("miniprogram")

    # Input
    parser_miniprogram.add_argument('-X', '--attribute_matrix', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
    parser_miniprogram.add_argument('-y', '--target_vector', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
    opts = parser.parse_args()
    opts_miniprogram = parser_miniprogram.parse_args()
    print(opts_miniprogram.__dict__)

if __name__ == "__main__":
    main()

Checking to make sure the docs work

# parent program
python parent_program.py --help
usage: parent_program.py [-h] {miniprogram} ...

optional arguments:
  -h, --help     show this help message and exit

subprograms:
  {miniprogram}

# miniprogram
python parent_program.py miniprogram --help
usage: parent_program.py miniprogram [-h] [-X ATTRIBUTE_MATRIX]
                                     [-y TARGET_VECTOR]

optional arguments:
  -h, --help            show this help message and exit
  -X ATTRIBUTE_MATRIX, --attribute_matrix ATTRIBUTE_MATRIX
                        Input: Path/to/Tab-separated-value.tsv
  -y TARGET_VECTOR, --target_vector TARGET_VECTOR
                        Input: Path/to/Tab-separated-value.tsv

Trying to run it:

python parent_program.py miniprogram -X ../../Data/X_iris.noise_100.tsv.gz -y ../../Data/y_iris.tsv
usage: parent_program.py miniprogram [-h] [-X ATTRIBUTE_MATRIX]
                                     [-y TARGET_VECTOR]
parent_program.py miniprogram: error: unrecognized arguments: miniprogram

Answer 1

The parent program could have code like

import mini_program
import sys
<do its own parsing>
if 'use_mini':
    <modify sys.argv>
    mini_program.main()

As written, importing mini_program doesn't run its parser. But calling its main will, but using the list it finds in sys.argv.

The parent parser should be written in a way that it accepts arguments that it needs, and doesn't choke on inputs the mini wants, '-X' and '-y'. It would then puts those 'extra' values in a modified sys.argv, which the mini parser can handle.

parse_known_args is one way of accepting unknown arguments, https://docs.python.org/3/library/argparse.html#partial-parsing

nargs=argparse.REMAINDER, https://docs.python.org/3/library/argparse.html#nargs, is another way of collecting remaining arguments for passing on.

If mini main was written as:

def main(argv=None):
    parser = argparse.ArgumentParser()
    parser.add_argument('-X', '--attribute_matrix', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
    parser.add_argument('-y', '--target_vector', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
    opts = parser.parse_args(argv)

it could be called with

mini_program.main(['-X', 'astring','-y','another'])

that is, with an explicit argv list, instead of working through sys.argv.

Keeping the main parser from responding to a '-h' help could be tricky. subparsers is probably the cleanest way of doing that.

You could combine subparsers with the invocation of a the mini main. I won't try to work out those details now.

Another way to define the main is:

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('-X', '--attribute_matrix', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
    parser.add_argument('-y', '--target_vector', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
    return parser

And use it as

 opts = main().parse_args()
 opts = mini_program.main().parse_args()

in other words, use main to define the parser, but delay the parsing.

Answer 2

My actual solution was an adaptation to the above:

# Controller
def main(argv=None):
    parser = argparse.ArgumentParser(prog="parent_program", add_help=True)
    parser.add_argument("subprogram")
    opts = parser.parse_args(argv)
    return opts.subprogram


# Initialize
if __name__ == "__main__":
    # Get the subprogram 
    subprogram = main([sys.argv[1]])
    module = importlib.import_module(subprogram)
    module.main(sys.argv[2:])