Combine base model with my Peft adapters to generate new model

Question

I am trying to merge my fine-tuned adapters to the base model. With this

torch.cuda.empty_cache()
del model
pre_trained_model_checkpoint = "databricks/dolly-v2-3b"
trained_model_chekpoint_output_folder = "/content/gdrive/MyDrive/AI/Adapters/myAdapter-dolly-v2-3b/"

base_model = AutoModelForCausalLM.from_pretrained(pre_trained_model_checkpoint,
                                  trust_remote_code=True,
                                  device_map="auto"
                                  )
model_to_merge = PeftModel.from_pretrained(base_model,trained_model_chekpoint_output_folder)
del base_model
torch.cuda.empty_cache()

merged_model = model_to_merge.merge_and_unload()

tokenizer = AutoTokenizer.from_pretrained(trained_model_chekpoint_output_folder)

Then

merged_model.save_pretrained('path')

The generated model size is the aproximatly the double. (5.6Gb to 11Gb) My fine tunning basically add info about 200 examples dataset in Alpaca format.

what am I doing wrong?

Answer 1

I guess your model was in torch.bfloat16 and you loaded it in 32 bits (which is the default).

just change

AutoModelForCausalLM.from_pretrained(pre_trained_model_checkpoint,
                                  trust_remote_code=True,
                                  device_map="auto"
                                  )

to

AutoModelForCausalLM.from_pretrained(
        pre_trained_model_checkpoint,
        trust_remote_code=True,
        return_dict=True,
        torch_dtype=torch.bfloat16,
        device_map="auto",
    )

Source: https://github.com/jondurbin/qlora/blob/main/merge.py