Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix the bug that _DataLoaderIterMultiProcess use time to generate the seed #43318

Merged
merged 2 commits into from
Jun 14, 2022

Conversation

zhangting2020
Copy link
Contributor

@zhangting2020 zhangting2020 commented Jun 8, 2022

PR types

Bug fixes

PR changes

APIs

Describe

fix the bug that DataLoaderIterMultiProcess use the time to generate seed

背景

#33310 中dataloader在多进程数据读取的场景下,会使用系统时间去生成随机种子。该PR是为了避免每个进程,以及每个epoch产生相同的随机数。

但使用系统时间生成随机种子,并且重置了numpy的随机种子,会导致在模型中即使固定numpy、random和paddle的随机种子后,训练结果依然无法复现。因为每一次启动训练,系统时间都不一样,原始的随机种子已经被dataloader重置。

PR效果

本PR在多进程数据读取中使用randint生成base_seed,替代使用系统时间生成base_seed的方式,可以解决 #33310 中提到的问题,并且在固定种子后多次运行,相同数据在经过dataloader和预处理(随机裁切、翻转等具有随机性的操作)后,可以得到稳定复现的输出。

  • 验证1:
    • 以下demo固定随机种子,固定原始数据,但在Dataset的getitem中引入一些随机性,模拟数据预处理的过程
    • DataLoader使用多进程数据读取(num_workers>0),shuffle=True
import numpy as np
import paddle
from paddle.io import Dataset, IterableDataset, BatchSampler, DataLoader
import random


seed = 300
paddle.seed(seed)
np.random.seed(seed)
random.seed(seed)

all_data = np.random.random((8, 3))
all_label = np.random.randint(0, 100, (8,))


class RandomDataset(Dataset):
    def __getitem__(self, index):
        data = all_data[index]
        label = all_label[index] * np.random.randint(0, 100, 1)
        return data, label

    def __len__(self):
        return 8

dataset = RandomDataset()
dataloader = DataLoader(dataset, batch_size=2, num_workers=2, shuffle=True)
epoch = 2
for i in range(epoch):
    print('epoch: ', i)
    for batch in dataloader:
        print(batch)
修复后输出
  • 第一次运行:

image

  • 第二次运行(可以看到尽管label预处理引入了随机性,但第二次运行和第一次的label是稳定复现的)

image

修复前的输出
  • 第一次运行:

image

  • 第二次运行:可以看到由于label的预处理引入的随机性,导致第二次运行和第一次的label已经不一样

image

@paddle-bot-old
Copy link

paddle-bot-old bot commented Jun 8, 2022

✅ This PR's description meets the template requirements!
Please wait for other CI results.

@paddle-bot-old
Copy link

paddle-bot-old bot commented Jun 8, 2022

你的PR提交成功,感谢你对开源项目的贡献!
请关注后续CI自动化测试结果,详情请参考Paddle-CI手册
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

heavengate
heavengate previously approved these changes Jun 8, 2022
Copy link
Contributor

@heavengate heavengate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Xreki
Xreki previously approved these changes Jun 8, 2022
Copy link
Contributor

@Xreki Xreki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhangting2020 zhangting2020 dismissed stale reviews from Xreki and heavengate via 0dd7083 June 13, 2022 02:19
Copy link
Contributor

@Xreki Xreki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhangting2020 zhangting2020 merged commit 2106f66 into PaddlePaddle:develop Jun 14, 2022
zhangting2020 added a commit to zhangting2020/Paddle that referenced this pull request Jun 21, 2022
… seed (PaddlePaddle#43318)

* fix the bug that _DataLoaderIterMultiProcess use time to generate the seed

* use np.random.randint to generate a base seed
lanxianghit pushed a commit that referenced this pull request Jun 22, 2022
… seed (#43318) (#43702)

 fix the bug that _DataLoaderIterMultiProcess use time to generate the seed

cherry-pick #43318
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants