How to write a config.json to train a Language model

225 Views Asked by At

Looking at NLP models on Huggingface I can see that each of those has a so called config.json file. Now I want to train a BART mode from scratch along this repo. To do so, I have to "write" a config.json myself. Now I can not find any documentation what attributes this config.json needs to contain.

The training script from the linked repo (./t5_train/t5_train.py) does fail if the provided config does not contain certain attributes. That way I found out what is missing but without docs I am clueless with what I have to set them.

Example what is among the expected attributes that I dont find a documentation for:

training:{
  device: ...,
  optimizer: ...,
  type: ...,
  noise: ...,
  noise_vocab: ...,
}
model:{
  src_lang: ...,
  trg_lang: ...,
  seq2seq: ...,
  dim_word_src: ...,
  n_layers_src: ...,
  bidirectional: ...,
}

Does somebody know where I can find a detailed documentation for how to write a config.json in particular and a config.json for BART in detail?


I allready took a look at https://huggingface.co/docs/transformers/main_classes/configuration but I could not find the attributes that the script expects.

0

There are 0 best solutions below