Luigi DictParameter KeyError

Question

I have the following task

class Test(luigi.Task):
    foo = luigi.DictParameter(default = {})
    ...

I am using a config file to run this

[Test]
foo = "{'a': 0, 'b': 1}"

I have also tried

• '{'a': 0, 'b': 1}'
• "{"a": 0, "b": 1}"
• "{'a': "0", 'b': "1"}"
• "{'a': 0, 'b': 1}"

They all fail with

KeyError: '"a"'

or

KeyError: ''a''

or some a related key error. What am I supposed to configure dictionaries as parameters when I am using a config file?

I tried this was as well but even this does not work. This link is from the PR that fixes the loading of dictionaries from TOML files.

In the official docs of the DictParameter they suggest

luigi --module my_tasks MyTask --tags

or

luigi --module my_tasks MyTask --tags '{"role": "web", "env": "staging"}'

But there is no mention for a TOML file. Even in their complex example TOML they do not include dictionaries

I have to say that Luigi is one of the most ill-documented projects I've ever worked with unfortunately.

Answer 1

I faced a similar issue while developing one of our projects. Unfortunately, we didn't find a solution that included dictionaries in luigi.cfg, instead we used a .yaml file and luigi.WrapperTask.

We did the following:

class BaseTask(luigi.WrapperTask):
  with open('/path/to/your/config.yaml') as f:
        foo_dict = yaml.safe_load(f)
  def requires(self):
    yield Test(foo=foo_dict)

class Test(luigi.Task):
 foo = luigi.DictParameter(default = {})

Hope it helps.

Answer 2

Here is what worked for me.

In your config file, do:

blah = "{'a':'b'}"

In your main file, you'd do

dict_param = luigi.DictParameter()

def run(self):
    parsed_dict_param = dict(eval(self.dict_param))

So the magic comes from eval.