how to fine-tune stable diffusion in sagemaker jumpstart with different descriptions per image

Question

I'm trying to fine-tune stable diffusion in sagemaker using this method: https://aws.amazon.com/blogs/machine-learning/fine-tune-text-to-image-stable-diffusion-models-with-amazon-sagemaker-jumpstart/

However that description and the example code uses a general single description for all images with the following instruction:

The dataset_info.json file must be of the format {'instance_prompt':<<instance_prompt>>}

However in my use case we need to use different descriptions, one per image, how can this be done?(currently already searched official documentation with no luck)

Answer 1

In the blog post you linked it states:

While fine-tuning, you may want to fine-tune on multiple subjects and have the fine-tuned model be able to generate images of all the subjects. Unfortunately, JumpStart is currently limited to training on a single subject. You can’t fine-tune the model on multiple subjects at the same time. Furthermore, fine-tuning the model for different subjects sequentially results in the model forgetting the first subject if the subjects are similar.