Numerous Tokens Performance

[Cpecific]

Performance on large files is atrocious.
You can look at code here https://github.com/Cpecific/twwh2_ancillary/blob/master/node_modules/tokenizr/lib/tokenizr.js
Mainly added Tokenizr._pendingPos. Switched from concatting and shifting arrays to simply iterating over them.
In my case it was an improvement from unknown amount of hours/days to barely a second. Probably issue crops up because of using transactions, but not sure, I wrote this code couple months back.
I would like to spend some time to improve it further, but sadly barely have time for such a side project.
Not sure if it is even the right way to do it, as I didn't try to understand all uses of library that much.
Probably have some bugs.

[rse]

Hmmm.... that's interesting. Sure, array operations might cost more than a simple integer operation, but I guess it mainly has to do in your case because of an excessive amount of backtracking with transactions and in this case the array operation costs accumulate dramatically. I'll check this in more detail when I find time again for the Tokenizr project.