Question about running the pretrain.py

[littleflow3r]

Hey,

I have difficulties in running the pretrain, any help would be appreciated.
So I've prepared corpus.txt (quite small, about 1000 lines) that looks like this:

document 1 line 1...
document 1 line 2...
document 1 line 3...

document 2 line 1...
document 2 line 2...
document 2 line 3...

And I run the pretrain.py but I got an error on train.py file, on this line:
print('Epoch %d/%d : Average Loss %5.3f'%(e+1, self.cfg.n_epochs, loss_sum/(i+1)))
So for the time being I commented that line.

And after I run again, here what I got:

Iter (loss=X.XXX): 0it [00:00, ?it/s]
Iter (loss=X.XXX): 0it [00:00, ?it/s]
Iter (loss=X.XXX): 0it [00:00, ?it/s]
Iter (loss=X.XXX): 0it [00:00, ?it/s]
Iter (loss=X.XXX): 0it [00:00, ?it/s]
Iter (loss=X.XXX): 0it [00:00, ?it/s]
Iter (loss=X.XXX): 0it [00:00, ?it/s]
Iter (loss=X.XXX): 0it [00:00, ?it/s]
Iter (loss=X.XXX): 0it [00:00, ?it/s]
....

Could you please point me where I could possibly make the mistake?
Thanks!

p.s. I have commented some part of the code in train.py (the part where it loads the checkpoint, because I dont install the tensorflow for a reason). What I want to do for now is training a pretrained bert model using my own data. I am not sure if it is causing the error above?

[Seoinsuk1]

same here

[Tikquuss]

I had the same problem as you.
But after I had checked I noticed that my dataset was empty (I'm talking about data_iter on line 219 of pretrain.py), because I specified the data in the wrong format.

My problem has been solved.