1 Answer
- Newest
- Most votes
- Most comments
1
Hi there,
instance_prompt is the prompt we would type to generate the image we are attempting to fine tune, while class_prompt denotes a prompt without the unique identifier/instance. This prompt is used for generating "class images" for prior preservation.
The dataset_info.json file is used in training to pass these arguments, and must be of format:
{
'instance_prompt':<<instance_prompt>>,
'class_prompt':<<class_prompt>>
}
So for example, if you're fine tuning the model with the face of a specific person, the dataset_info.json file could look like this:
{
"instance_prompt": "a photo of a zwx person",
"class_prompt": "a photo of a person"
}
Where 'zwx' would be the unique identifier that is used to train the model.
See here for more details.
Hope this helps
answered 9 months ago
Relevant content
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
can you tell me how to write for multiple images
{ "instance_prompt": "a photo of a zwx person", "class_prompt": "a photo of a person" }
The group of images is used to train the model on the same instance prompt, not one prompt per image.
what about if we want to train the model on different variety of images and provide different prompt for different categories/ type of images
{ "instance_prompt": { "coco_set.jpg": "wooden shelf with modern style", "coco_sets_with_two_mirror.jpg": "wooden shelf with modern style", "maserati_black.jpg": "The Maserati chair with the black color is perfect for the study room", "maserati_gray.jpg": "The Maserati chair with the gray color is perfect for the living room", "maserati_white.jpg": "The Maserati chair with the white color is perfect for the living room", "panda_left_facing.jpg": "x chair with left facing with white color.", "panda_orange_chair.jpg": "hallowing chair with orange color is perfect for modern style.", "panda_white_chair.jpg": "wooden chair with white color is perfect for modern style.", "toronto_bedstead.jpg": "grey-bedstead with minimalistic style.", "toronto_bedstead1.jpg": "bedstead with minimalistic style." } } is this way correct?
how to train multiple category or types of images and with different prompts
No, it's not correct. In the dataset_info.json file you must pass a single instance_prompt and class_prompt for all the images in the input directory, as you need more than one image to fine-tune the model on one subject. Moreover, you should specify a unique identifier in the instance prompt for the model to be able to distinguish it. Your dataset_info.json file should look similar to the one included in the answer as example.
Please refer to this blogpost (https://aws.amazon.com/blogs/machine-learning/fine-tune-text-to-image-stable-diffusion-models-with-amazon-sagemaker-jumpstart/) and DreamBooth paper (https://dreambooth.github.io/) for a more in-depth explanation.
As explained in the blogpost, if you are using JumpStart, it is currently limited to training on a single subject. You can’t fine-tune the model on multiple subjects at the same time. Furthermore, fine-tuning the model for different subjects sequentially results in the model forgetting the first subject if the subjects are similar. To achieve training on multiple subjects, you should:
Hello jnavrro,
I would like to express my gratitude for your assistance. Your guidance has been truly valuable.
I'm currently seeking alternative methods to train a stable diffusion model across numerous images and categories, similar to the approach used in Dreambooth. I'm curious if this can be achieved through platforms like Jumpstart or SageMaker. Additionally, I'm interested in exploring the possibility of implementing the Dreambooth approach within the SageMaker framework for training on multiple images.