I use Diffusers to train LoRA. Training images are my photos, but the result image is not like me

Question

Here is my training code.

from accelerate.utils import write_basic_config
write_basic_config()

import os

os.environ["MODEL_NAME"] = "runwayml/stable-diffusion-v1-5"
os.environ["INSTANCE_DIR"] = "/notebooks/me_photos"
os.environ["OUTPUT_DIR"] = "/notebooks/me_model_1_22"
script_path = "/notebooks/diffusers/examples/dreambooth/train_dreambooth_lora.py"

!accelerate launch {script_path} \
  --pretrained_model_name_or_path={os.environ["MODEL_NAME"]} \
  --instance_data_dir={os.environ["INSTANCE_DIR"]} \
  --output_dir={os.environ["OUTPUT_DIR"]} \
  --instance_prompt="a photo of Ryan" \
  --resolution=512 \
  --train_batch_size=1 \
  --learning_rate=2e-6 \
  --max_train_steps=2400 \
  --gradient_checkpointing \
  --use_8bit_adam \
  --with_prior_preservation \
  --prior_loss_weight=1.0 \
  --class_data_dir="/notebooks/faces_prior_preservation" \
  --class_prompt="a photo of person face"

Here is the code to produce image.

from diffusers import DiffusionPipeline
import torch

# Initialize logging
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()


pipe = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", 
                                         torch_dtype=torch.float16, 
                                         use_safetensors=True, 
                                         variant="fp16")
pipe.to("cuda")


pipe.load_lora_weights("/notebooks/me_model_1_22", weight_name="pytorch_lora_weights.safetensors" , adapter_name="me")

active_adapters = pipe.get_active_adapters()
active_adapters

logger.info(f"LoRA {active_adapters} loaded successfully.")


# Generate an image
prompt = "a photo of Ryan"
lora_scale= 1
image = pipe(
    prompt, num_inference_steps=30, cross_attention_kwargs={"scale": lora_scale}
).images[0]

# Save the image
output_path = "/notebooks/image_of_me2.png"
image.save(output_path)
logger.info(f"Image saved at {output_path}")

# Clean up
del image
torch.cuda.empty_cache()

Answer 1

I believe you might not have used a good dataset of images with good captions and an unique token. Ryan looks like it might be something know to the model from before. Also one way to check its effect would be to generate two Images with same seed, one with LoRA and one without.