How to reset parameters from AutoModelForSequenceClassification?

77 Views Asked by At

Currently to reinitialize a model for AutoModelForSequenceClassification, we can do this:

from transformers import AutoModel, AutoConfig, AutoModelForSequenceClassification

m = "moussaKam/frugalscore_tiny_bert-base_bert-score"
config = AutoConfig.from_pretrained(m)
model_from_scratch = AutoModel(config)

model_from_scratch.save_pretrained("frugalscore_tiny_bert-from_scratch")

model = AutoModelForSequenceClassification(
  "frugalscore_tiny_bert-from_scratch", local_files_only=True
)

Is there some way to reinitialize the model weights without saving a new pretrained model initialized with AutoConfig?

model = AutoModelForSequenceClassification(
  "moussaKam/frugalscore_tiny_bert-base_bert-score", 
  local_files_only=True
  reinitialize_weights=True
)

or something like:

model = AutoModelForSequenceClassification(
  "moussaKam/frugalscore_tiny_bert-base_bert-score", 
  local_files_only=True
)

model.reinitialize_parameters()
1

There are 1 best solutions below

1
On BEST ANSWER

That is the purpose of from_config (i.e. creating a model but not loading the respective weights):

from transformers import AutoModel, AutoConfig, AutoModelForSequenceClassification

m = "moussaKam/frugalscore_tiny_bert-base_bert-score"
config = AutoConfig.from_pretrained(m)

no_weights_model = AutoModelForSequenceClassification.from_config(config)
weights_model = AutoModelForSequenceClassification.from_pretrained(m)

import torch

print(torch.allclose(no_weights_model.bert.embeddings.word_embeddings.weight,weights_model.bert.embeddings.word_embeddings.weight))

Output:

False