I trained a DonUT model to perform document extraction from receipt, after the training I tested the accuracy and everything was fine so I pushed the tokenizer and the model to hub.
But when I call it from hub the model behave as if it was never trained, the results are not even coherent. This happens only with the peft finetuned model. I have done a full finetuning just before and everything is fine.