How to load a .py script directly with evaluate.load?

Question

If I have a script like this https://huggingface.co/spaces/evaluate-metric/frugalscore/blob/main/frugalscore.py and save it as fgscore.py with a directory locally like:

./
  my_script.py
  fgscore/
      fgscore.py

And in my_script.py, I can do something like:

import evaluate


mt_metrics = evaluate.load("fgscore")

sources = ["안녕하세요 저는 당신의 아버지입니다", "일반 케노비"]
predictions = ["hello here I am your father", "general kenobi"]
references = ["hello there I am your father", "general yoda"]
results = mt_metrics.compute(predictions=predictions,
                             references=references)

print(results)

Looking at the evaluate.load() function, https://github.com/huggingface/evaluate/blob/main/src/evaluate/loading.py#L688, it states:

path (`str`):
    Path to the evaluation processing script with the evaluation builder. Can be either:
        - a local path to processing script or the directory containing the script (if the script has the same name as the directory),
            e.g. `'./metrics/rouge'` or `'./metrics/rouge/rouge.py'`
        - a evaluation module identifier on the HuggingFace evaluate repo e.g. `'rouge'` or `'bleu'` that are in either `'metrics/'`,
            `'comparisons/'`, or `'measurements/'` depending on the provided `module_type`

Is there a reason to do {name}/{name}.py for using the path arguments in evaluate.load()?

Is there a way to override such that I can point the evaluate directly to the .py file? E.g. evaluate.load("fgscore.py")

Answer 1

There's no need to repeat the name and a path when using evaluate.load(), you can have the directory structure like this:

./
  my_script.py
  fgscore.py

And in the my_script.py do this:

import evaluate


mt_metrics = evaluate.load("./fgscore.py")

sources = ["안녕하세요 저는 당신의 아버지입니다", "일반 케노비"]
predictions = ["hello here I am your father", "general kenobi"]
references = ["hello there I am your father", "general yoda"]
results = mt_metrics.compute(predictions=predictions,
                             references=references)

print(results)