Tokens-database refactoring

[muz]

I had the idea of poking at systematically finding and generating tokens as new set data is released. To enable this, some understanding of the current database was needed on my side. I then saw // TODO: remove "generate" from db on the codebase and some codepaths that could be tidied up a bit...

This should be a functional no-op to the end user, but internally:

• Cleans up and addresses that TODO, removing that unnecessary column
• Adds some documentation on the database schema
• Validates that items in the tokens-database.txt are as expected
• Given that we're validating more strictly with a fairly tight regex, we can safely extract and clean up tokenType and setCode extraction
• Also removes the no-longer user ImageFileName column

I was able to confirm that the existing data in database was consistently formatted with the same number of columns;

muz@m1 resources % cat tokens-database.txt | grep -e '^|' | sed -e 's/[a-zA-Z0-9: ,\-]//g' -e "s/'//" | sort | uniq
|||||||

I was also able to confirm what used to be the ImageFileName column was empty for all rows;

muz@m1 resources % cat tokens-database.txt | grep -e '^|'  | awk -F '|' '{print $6}' | sort | uniq 

Confirmed that the only thing reading the tokens-database.txt and in active use right now is TokenRepository.java with objects it extracts being consumed downstream in the project. The generated objects from the Repository should be consistent with before.

I did note that there is a Unit Test for this class, but it's currently marked as @Ignore with // TODO: enable test after massive token fixes so opted to not touch this right now if the test itself is not exercising value. The changes it's expecting are well beyond the scope of what this PR seeks to achieve on closing out that aforementioned TODO

[muz]

This needs rebasing given the flurry of changes to support ECC and ECL in recent weeks

[muz]

Rebased and tested with a locally built client that token data is loaded correctly and works in-game as expected.

I think this is ready for review to ultimately address the TODO on deprecating Generate. If this is merged, I can also have a stab at updating the Wiki documentation