|
self.val_dataset = RLHFDataset(parquet_files=self.config.data.val_files, |
|
tokenizer=self.tokenizer, |
|
prompt_key=self.config.data.prompt_key, |
|
max_prompt_length=self.config.data.max_prompt_length, |
|
filter_prompts=True, |
|
return_raw_chat=self.config.data.get('return_raw_chat', False), |
|
truncation='error') |
|
self.val_dataloader = DataLoader(dataset=self.val_dataset, |
|
batch_size=len(self.val_dataset), |
|
shuffle=True, |
|
drop_last=True, |
|
collate_fn=collate_fn) |
It seems that val_batch_size is not utilized. It would always be the length of the RLHFDataset, which is length of the DataFrame?
