Overview

This post includes resources for training efficient LoRA models for fine-tuning Stable Diffusion. We'll discuss preparing training data by focusing on key tags and concepts, cleaning images, and automating tagging. We also provide some recommended training settings for character LoRA models. Lastly, we provide a formula and calculator for determining total training steps based on variables like image count, repeats, batch size, and epochs.

Resources

Check out the resources below to learn how to train your own LoRA model.

Videos

Other Guides

[Guide] Make your own Loras, easy and free

Stable Diffusion LoRA training science and notes

Make a train.ps1 Powershell script for the Kohya repo

LoRA Training Guide

Lazy Training Guide

LoRA Dreambooth Guide (English)

Tips & Tricks to Prepare a Dataset for Training

Tip #1 - Captions can be different depending on the base model

• If you are training a LoRa based on an anime model (NAI, AnythingV3), make the description strictly in the style of Danbooru/Gelbooru tags. For example, 1girl, short hair, green eyes, black hair, school uniform....

• If you are using SD 1.x/2.x models, write what you see in the image for the captions. For example, if the image is a picture of a bearded fisherman at work, write pencil art of a man fishing, beard; if it is a picture of your smiling friend wearing glasses, red shirt, sunset in the background, write photo of friend_name, smiling, wearing glasses, red shirt, sunset in the background.

Tip #2 - Focus on concept words

• If you are training a character/person, it is better if the dataset consists only of the images where he is present. And all captions should have a keyword describing this character/person.

For example,

1.txt: 

photo of JohnFriend, jacket, jeans

2.txt: 

photo of JohnFriend, shorts, t-shirt.

You will use the tag JohnFriend to call this person in the prompt.

• The same works for anime models - in this case it will be the name of the character.

For example,

1girl, shiina mayuri, short hair, green eyes, etc.

Call out with shiina mayuri, respectively.

• If you are training an author's style, as in the previous paragraph, it is advisable that the dataset consist only of images drawn by this author.

Example of caption file: ArtistName, mountains, night, moon, snowy peaks, stars, etc. Calling the artist's style via ArtistName.

Tip #3 - It is better to clean up your images

All the bits of other people's limbs, strange objects in the picture, artists' signatures, links – anything you don't want to see in the generation is better trimmed or masked in image editor.

Tip #4 - Speed up the tagging process

Make the machines do the hard work.

• To speed up manual tagging you can use sd-tagging-helper and BooruDatasetTagManager.
• There are also extensions (1, 2, 3) for A1111-webui.You can use Grabber with these settings (copy and paste: %all:unsafe,separator=^,^ %) to download images from various booru boards along with tags.

Troubleshooting for Kohya_ss_GUI

Problem likely causes

• Due to using Configuration File from the Youtube Channel

Due to misconfig of fp16 and bp16 → choose fp16 as the installation process instruct to use fp16

• Due to “8bit adam” in the Advanced Configuration Tab under Training Parameters. → unchecked it

Learn more here.

Recommended Training Settings

Some of the videos or guides above may include settings files that you can simply load to get the same settings all the time. However, here’s some recommended settings if you are using a character or person as a model for LoRA training:

• Image training loop - this is the number that goes before your model/subject (e.g. XX_nameofperson) = 10_
• Batch size: 6
• Epoch: 14
• Precision: bf16
• 512,512
• Scheduler: Polynomial
• Network Rank: 160
• Network Alpha: 160
• Learning Rate: Default: 0.0001
• Text Encoder Learning Rate: 0.00005
• LR warmup: 10
• Optimizer: AdamW8bit

Note Regarding Epochs

Epochs can be used to generate snapshots of your LORA during training. What this means is that if you set your epochs to 8, and then save every 1 epoch, you will create 8 versions of your LORA. This is obviously fantastic for testing purposes, and with the introduction of the newer script and faster epochs, you can go as high as you want. Obviously, this will mean that it will bake your LORA up to the final epoch, and it will take longer, so keep in mind that you probably don't need to go over 4-6 unless you're baking a higher fidelity LORA or a concept.

Use these settings on the script. Simply copy and paste over the default settings, ensuring that you copy over the correct settings. Remember to amend the file paths. It is my understanding that you can change 'batch size' up or down depending on your VRAM with no noticeable impact on the final generation, but this has not been extensively tested.

Steps Formula & Calculator

Formula

You can use the formula below to determine Batch Size, Epoch, Image Repeats, or Total Steps. The training images number is usually known, based on how many images you have.

So you could arbitrarily play around with the other numbers depending on how many epochs you want, or the batch size, or image repeats.

Total Steps = (Image Repeats / Batch Size) * Training Images * Epoch

Lora Training Calculator

Below are some examples for determining the total steps you would have, depending on the image repeats, batch size, training images, and epoch.