[SD3] the training script of SD3 dreambooth has wrong logit-normal weighting

[Luciennnnnnn]

Describe the bug

I guess the implementation of logit-normal weighting is wrong

diffusers/examples/dreambooth/train_dreambooth_lora_sd3.py

Line 1490 in 896fb6d

u = torch.normal(mean=args.logit_mean, std=args.logit_std, size=(bsz,), device=accelerator.device)

.

Consider that timestep is sampled uniformly first, weighting function should be a function that depends on pre-sampled timestep, however current version is just a random sample of logit-normal distribution.

Reproduction

no

Logs

No response

System Info

no

Who can help?

@DN6 @yiyixuxu @sayakpaul @DN6 cc.

[sayakpaul]

Fixed by #8528. Cc: @kashif

[feng-yufei]

Request: can anyone confirm that the default weighting sigma_sqrt corresponds to the uniform weighting? After checking with the training objective here (#8567) and the current implementationnoisy_model_input = sigmas * noise + (1.0 - sigmas) * model_input and model_pred = model_pred * (-sigmas) + noisy_model_input I found the training loss

diffusers/examples/dreambooth/train_dreambooth_lora_sd3.py

Line 1530 in eeb7003

(weighting.float() * (model_pred.float() - target.float()) ** 2).reshape(target.shape[0], -1),

has an additional sigma multiplier, which cancels with the sigma**-2.0, so I wonder whether all the reweighting should be based on the sigma_sqrt first?

[sayakpaul]

^ seems like a different issue than the original question. So, will close this issue.