OpenMDAO hierarchical solvers recording

Question

In OpenMDAO, is there any recommendation on how to record and read solver cases if the model is composed of multiple groups/cycles, and multiple nonlinear solvers?

I have a model built of 2 cycles (cycle1 and cycle2), one of them containing two subcycles (cycle1_1 and cycle1_2). For now I am attaching a solver to each of my nonlinear solvers:

solver1 = model.cycle1.nonlinear_solver
solver1_1 = model.cycle1.cycle1_1.nonlinear_solver
solver1_2 = model.cycle1.cycle1_2.nonlinear_solver
solver2 = model.cycle2.nonlinear_solver

solver1.add_recorder(recorder)
solver1_1.add_recorder(recorder)
solver1_2.add_recorder(recorder)
solver2.add_recorder(recorder)

When trying to read the results with:

cr = om.CaseReader(results)

I am getting the following error:

RuntimeError: Can't parse solver iteration coordinate: rank0:root._solve_nonlinear|0|NLRunOnce|0|cycle1._solve_nonlinear|0|NonlinearBlockGS|1|cycle1.cycle1_1._solve_nonlinear|1|NonlinearBlockGS|1

I am looking to get information about the convergence history and some plots on the coupling variables.

EDIT: My code has a structure similar to that in https://openmdao.org/newdocs/versions/latest/basic_user_guide/multidisciplinary_optimization/sellar.html, with the groups defined in setup:

import openmdao.api as om

class MDA(om.Group):
    
    # class ObjCmp(om.ExplicitComponent):
        # some objective component
    # class ConCmp(om.ExplicitComponent):
        # some constraint component
    
    def setup(self):
        
        cycle1 = self.add_subsystem('cycle1', om.Group(), promotes=['*'])
        
        cycle1_1 = cycle1.add_subsystem('cycle1_1', om.Group(), promotes=['*'])
        cycle1_1_comp = cycle1_1.add_subsystem('comp', om.ExecComp('x1 = 3 + x2'), promotes=["*"])

        cycle1_2 = cycle1.add_subsystem('cycle1_2', om.Group(), promotes=['*'])
        cycle1_2_comp = cycle1_2.add_subsystem('comp',  om.ExecComp('x2 = 3 + x1 + y'), promotes=["*"])

        cycle2 = self.add_subsystem('cycle2', om.Group(), promotes=['*'])
        cycle2.add_subsystem('comp', om.ExecComp('y = x1 + 2'), promotes=['*'])

p = om.Problem(model=MDA())
model = p.model

p.setup()

p.run_model()

Answer 1

Unfortunately, as of OpenMDAO V3.16 this looks like a bug. Its been logged as a high priority issue on the OpenMDAO development backlog: Issue #2453

I can replicate it with the following script:

import openmdao.api as om

p = om.Problem()

model = p.model

cycle1 = p.model.add_subsystem('cycle1', om.Group(), promotes=['*'])
cycle1_1 = cycle1.add_subsystem('cycle1_1', om.Group(), promotes=['*'])
cycle1_1_comp = cycle1_1.add_subsystem('comp', om.ExecComp('x1 = 3 + x2'), promotes=["*"])

cycle1_2 = cycle1.add_subsystem('cycle1_2', om.Group(), promotes=['*'])
cycle1_2_comp = cycle1_2.add_subsystem('comp',  om.ExecComp('x2 = 3 + x1 + y'), promotes=["*"])

cycle2 = p.model.add_subsystem('cycle2', om.Group(), promotes=['*'])
cycle2.add_subsystem('comp', om.ExecComp('y = x1 + 2'), promotes=['*'])


solver1 = model.cycle1.nonlinear_solver
solver1_1 = model.cycle1.cycle1_1.nonlinear_solver
solver1_2 = model.cycle1.cycle1_2.nonlinear_solver
solver2 = model.cycle2.nonlinear_solver

print(solver1, solver1_1, solver1_2, solver2)

recorder = om.SqliteRecorder('cases.db')

solver1.add_recorder(recorder)

# recorders on nested solvers trigger the bug 
# solver1_1.add_recorder(recorder)
# solver1_2.add_recorder(recorder)

# Kind-of workaround, put the recorder on the child component/group instead
cycle1_1_comp.add_recorder(recorder)
cycle1_2_comp.add_recorder(recorder)

solver2.add_recorder(recorder)

p.setup()

p.run_model()


reader = om.CaseReader('cases.db')

print(reader.list_sources())

It seems to be the nested recorders that are triggering the bug. As a kind-of workaround you can stick the recorder on the lower level group/component instead. That will make it a bit harder to know which cases are from which solver iteration, but the naming scheme of the iteration coordinates should at least help a little there. Hopefully that gets you moving in the meantime, while the bug is fixed.