comparing array_record's ArrayRecordDataSource + grain data loader to huggingface. (Grain is extremely slow)

[ZurabDz]

Hello,

I have opened up similar issue in array_record's github page, and in terms of array_record perfomance reading like dryman suggested is right. The problem is how can I use grain as a data loader and sampler, if using __getitem__ is so slow on which ArrayRecordDataSource depends on?. Any ideas what I am doing wrong?