I've been performing classification using GPT-3/3.5/4 models by restricting outputs using the logit_bias parameter. I am not sure how to do the same in open source models, specifically llama, llama2, and their derivatives.
I have the model weights for llama and llama2, but I have not been approved for llama2 HuggingFace. Approval is taking too long so I think I am going to use the script provided by meta itself. I do not see any logit_bias parameter in the generation function in llama models.
Could someone point out how to specify logit_bias for llama 1 and 2 models using the meta scripts?
I see Hugging Face has a way to use logit_bias here. I have not tried it yet, I hope I can use this for llama derivatives in Hugging Face. I am unable to use the conversion script from Hugging Face to turn my meta llama weights to hg version due to memory constraints and running on a remote server using ssh.