I'm trying to use the checkpoint from a finetuned gpt-2-simple to use in my modified version of this gpt2 output detector https://colab.research.google.com/drive/11OZknA7er__NgH-Yje7__LY5v-huYE0P?usp=share_link
My saved checkpoint has the following files: encoder.json, hparams.json, model-1000.data-00000-of-00001, model-1000.index, model-1000.meta, vocab.bpe
I'm trying to load this to export it to onnx and then to use the detector.
I have tried:
config = transformers.GPT2Config.from_pretrained('/content/hparams.json')
tokenizer = transformers.GPT2Tokenizer("/content/encoder.json", "/content/vocab.bpe")
model = transformers.GPT2Model.from_pretrained('/content/model.index',from_tf=True,config=config)
But I get error: OpError: /content/model.data-00000-of-00001; No such file or directory
In the original collab they load the checkpoint directly from gpt2 so I can't use the same process.
I understand it's complaining it doesn't have the model-1000.data-00000-of-00001 file. What I don't understand and haven't been able to find is how to load it into this model.
I have seen these two StackOverflow answers that explain what the files are but not how to load a checkpoint into a model.
What do the .index and .data-00000-of-00001 files produced after saving in TensorFlow represent? TensorFlow, why there are 3 files after saving the model?
Where is the.data-00000-of-00001 file supposed to be loaded and how? Is this possible?