python, hydra, load variable from yaml to list

Question

I got a below code:

# settings.yaml

my_class:
  columns_id: ${oc.env:MY_LIST}
  all_columns: {"all_columns_as_str": ${oc.env:MY_LIST}}

where my variable is MY_LIST=a,b,c

then I got:

# hydra_conf.py

from dataclasses import dataclass

@dataclass
class MyClass:
  columns_id: list
  all_columns: dict[str, str]

and then I use it all like below:

# main.py
import hydra
from hydra_conf import MyClass

@hydra.main(config_path="settings", config_name="settings")
def main(cfg: MyClass) -> None:
    assert isinstance(cfg.columns_id, list)
    assert isinstance(cfg.all_columns, dict)
    assert cfg.all_columns['all_columns_as_str'] == 'a,b,c'

How I need to configure my yml.file or my hydra_conf.py file to load columns_id as list and all_columns as a dict[str, str]?

Answer 1

Give this a try:

# settings.yaml
defaults:
  - my_class_schema
  - _self_

columns_id: ${oc.decode:"[${oc.env:MY_LIST}]"}
all_columns:
  all_columns_as_str: ${oc.env:MY_LIST}

# hydra_conf.py
from dataclasses import dataclass
from typing import Any
from hydra.core.config_store import ConfigStoreWithProvider

@dataclass
class MyClass:
    columns_id: list[Any]
    all_columns: dict[str, str]

cs = ConfigStoreWithProvider(__file__)
cs.store("my_class_schema", MyClass)

# main.py
import hydra
from omegaconf import DictConfig, OmegaConf
from hydra_conf import MyClass

@hydra.main(config_path=".", config_name="settings", version_base=None)
def main(cfg_: DictConfig) -> None:
    cfg: MyClass = OmegaConf.to_object(cfg_)
    assert isinstance(cfg, MyClass)
    assert isinstance(cfg.columns_id, list), f"{cfg.columns_id}"
    assert isinstance(cfg.all_columns, dict), f"{cfg.all_columns}"
    assert cfg.all_columns["all_columns_as_str"] == "a,b,c"

if __name__ == "__main__":
    main()

Here's how this works:

• Including my_class_schema in the defaults list means that your yaml file will be merged with the schema defined by MyClass. See the Hydra docs on Structured Configs (and specifically on using a Structured Config Schema) for more info about this.
• The string "[${oc.env:MY_LIST}]" results in "[a,b,c]" after the interpolation is resolved.
• The oc.decode resolver parses the string "[a,b,c]", resulting in ["a", "b", "c"].