How can I organize Hydra 1.3 configuration files and import them into each other?

Question

I am trying to use Hydra 1.3 contrive a simple, but apparently not trivial, configuration that maps endpoints of a given API to their corresponding processing functions.

So far, I came up with a config folder structure that looks like:

$ tree conf
conf
├── api
│   └── api_1.yaml
│   └── api_2.yaml
├── endpoints
│   ├── end_1.yaml
│   ├── end_2.yaml
│   └── end_3.yaml
├── models
│   ├── model_1.yaml
│   ├── model_2.yaml
│   └── model_3.yaml
└── secret
│   └── manager.yaml
└── config.yaml
└── config_e.yaml
└── config_t.yaml

Where

# api_1.yaml

app: app_one
base_url: "https://base_url.com/"
auth:
id_token:
client_id: client_1_id
client_username: client_1_id_username
client_password: client_1_id_password
token: null
auth_url: "https://auth_url.com/"
endpoints:
  - endp_1: ${end_1.endpoint_1}
  - endp_3: ${end_3.endpoint_3}

# api_2.yaml

app: app_two
base_url: "https://base_url.com/"
auth:
id_token:
client_id: client_2_id
client_username: client_2_id_username
client_password: client_2_id_password
token: null
auth_url: "https://auth_url.com/"
endpoints:
  - endp_2: ${end_2.endpoint_2}

# end_1.yaml

name: endpoint_1
method: POST
payload:
  att_1: "string"
  att_2: "string"

# end_2.yaml

name: endpoint_2
method: GET
payload:
  att_1: "string"
  att_2: "string"

# end_3.yaml

name: endpoint_3
method: POST
payload:
  att_1: "string"
  att_2: "string"

# model_1.yaml

model:
  class_name: app.Model1
  id: model_1_id
  name:
    table: 'model1'
  fields:
    attr_1: att_1
    attr_2: att_2
  dependants:
    model2:
        obj: ${model_2}
        key: attr_1
    model3:
        obj: ${model_3}
        key: attr_1

# model_2.yaml

model:
  class_name: app.Model2
  id: model_2_id
  name:
    table: 'model2'
  fields:
    attr_1: att_1
    attr_2: att_2
  dependants:
    null

# model_1.yaml

model:
  class_name: app.Model3
  id: model_3_id
  name:
    table: 'model3'
  fields:
    attr_1: att_1
    attr_2: att_2
  dependants:
    null

# config_e.yaml

defaults:
  apis:
  - api_1: api_1
  - api_2: api_2
  secret: manager

# config_t.yaml

defaults:
  - model1: model_1
  - model2: model_2

Based on the getting started, I came up with the snippet shows how I the configs are to be used.

import hydra
import requests
from omegaconf import DictConfig, OmegaConf

@hydra.main(version_base=None, config_path="conf", config_name="config_e")
def my_app(cfg : DictConfig) -> None:

    for api in cfg.apis:
    for endpoint in api.endpoints:
        response = requests.post(api.base_url, payload=endpoint.payload)

if __name__ == "__main__":
    my_app()

I am struggling with the syntax, for instance, in the apis' yamls:

endpoints:
  - endp_2: ${end_2.endpoint_2}

Shouldn't it turn api_2.yaml into:

# api_2.yaml

app: app_two
base_url: "https://base_url.com/"
auth:
id_token:
client_id: client_2_id
client_username: client_2_id_username
client_password: client_2_id_password
token: null
auth_url: "https://auth_url.com/"
endpoints:
  - endp_2: 
    name: endpoint_2
    method: GET
    payload:
      att_1: "string"
      att_2: "string"

and analogously model_1.yaml's dependants objs into the corresponding models?

How can I: 1 - organize the folder structure to make it easier to manage. 2 - correctly name the parameters 3 - import a whole config into another one's attributes

As it seems, I am rather confused with the syntax, even after having read the docs.

NOTE: The api's endp_1 and the config_t's model1 do not match the file name, and this is because I may need to rename them.

Answer 1

The following syntax is interpolation. You can learn about it in the OmegaConf documentaton.

endpoints:
  - endp_2: ${end_2.endpoint_2}

Interpolation does not "turn" the YAML into anything. You can think of it as a pointer in the config that is resolved when you are accessing it. In the above snippet, there is no node matching end_2.endpoint_2 so accessing it at runtime would result in an exception. For this to work, you would need something like this in your config:

endpoints:
  - endp_2: ${end_2.endpoint_2}

end_2:
  endpoint_2: SOMETHING

When accessing endpoints[0].endp_2, you would get SOMETHING. You can print the OmegaConf config object while resolving all interpolations to know what the config actually looks like. See OmegaConf.to_yaml() and pass resolve=True. Note that this will fail if an interpolation cannot be resolved.

This:

defaults:
  apis:
  - api_1: api_1
  - api_2: api_2
  secret: manager

Is NOT a valid Defaults List in Hydra. There is no nesting in the Defaults List.