[Wav2Vec2] Improve SpecAugment function by converting numpy based fun…

[punitvara]

…ction to pytorch based function

Implements #10459

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

[patrickvonplaten]

Let's try to not use np here. If we use torch.from_numpy(...) - it's not GPU-friendly

[punitvara]

mask_idc = torch.tensor([mask_idc[j] + offset for j in range(len(mask_idc)) for offset in range(lengths[j])])

I just did this. Is this right ?

[patrickvonplaten]

Let's try to get rid of the for-loop and do tensor operations only

[punitvara]

I am not sure how to do this. Something like following but not getting how should I put it ? Can you help here.

mask[i, mask_idc] = [True, torch.randperm(mask_idc)[:min_len] if torch.tensor(mask_idcs).size() > min_len]

[patrickvonplaten]

We need to run benchmark tests to see by how much the speed improved both on CPU and GPU

[punitvara]

@patrickvonplaten Can you please help with above comments ?

[patrickvonplaten]

Hey @punitvara,

At the moment, I sadly don't have the time to handle the big chunk of the PR. It would be great if you could try to:

1. Find a way to benchmark your new function on GPU and show that it yields a speed-up in the forward pass compared to the old function

2. Try out some advanced PyTorch indexing to replace the for loops.

Taking a look at those PRs should help you: #9600, #9453, #6064

[patrickvonplaten]

Closing due to inactivity. Sorry @punitvara, I saw a lot of interest from other people to open a PR and this one seems to have stalled. Feel free to re-open it and give it a second shot if you want :-)

[punitvara]

I got busy into some other work. I will try to work on different issue. If you get any PR, feel free to merge it