Tokenizer won't load from Huggingface hub

145 Views Asked by At

error screenshot

from transformers import AutoConfig, AutoTokenizer, AutoModelForSequenceClassification, AutoModelForCausalLM, AutoModelForSeq2SeqLM
from transformers import GPT2Tokenizer
tok1 = AutoTokenizer.from_pretrained(r"C:\Users\user\Documents\learn\hf_models\gpt2") # It works
tok2 = AutoTokenizer.from_pretrained('gpt2') # It takes infinitely long time

When I am loading from GPT2 model from Huggingface Hub, it won't load and takes infinitely long time. The screenshot shows 1m 32.9secs but it actually took more than an hour and won't finish.

If I download the necessary files in my PC and load it as in the 3rd line of the code (tok1), it works. But if I download from the Huggingface Hub directly as in the 4th line of the code (tok2), it does not work.

What I have tried so far is...

  1. To delete everything remaining in 'C:\Users\user.cache\huggingface\hub' --> It did not work
  2. To run the same code on Google Colab --> It works but I want to do it on my local machine.
  3. Trying other models than GPT2 such as distilgpt2 --> It didn't work either.

The environment is Windows 11, Python 3.11.0, Visual Studio Code 1.79.2.

0

There are 0 best solutions below