How to specify additional parameters when using HuggingFace Evaluate's evaluate.combine() method?

32 Views Asked by At

I am using the HuggingFace Evaluate library to evaluate my results using 2 metrics. Here are the codes:

import evaluate

metric = evaluate.combine(
    ["sacrebleu", "chrf"], force_prefix=True
)

And in the compute_metrics() function, here is how I call the metric.compute():

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)

    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels)
    
    results = {"bleu": result["sacrebleu_score"], "chrf": result["chr_f_score"]}

    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    results["gen_len"] = np.mean(prediction_lens)
    results = {k: round(v, 4) for k, v in results.items()}
    return results

However, I would like to specify the chrF to use word_order=2. How can I do so? Thanks.

0

There are 0 best solutions below