test tokenize json fail

[MXueguang]

/home/xueguang/anaconda3/envs/pyserini_head/bin/python /home/xueguang/.local/share/JetBrains/PyCharm2020.3/python/helpers/pycharm/_jb_unittest_runner.py --target test_tokenize_json.TestTokenizeJson
Testing started at 8:20 p.m. ...
Launching unittests with arguments python -m unittest test_tokenize_json.TestTokenizeJson in /home/xueguang/Research/pyserini/tests

Converted 0 docs, writing into file ./test_out_tokenize_json/docs00.json
Converted 0 docs, writing into file ./test_out_tokenize_json/docs01.json


a new gp ##u ! != i have a new gp ##u !

Expected :i have a new gp ##u !
Actual   :a new gp ##u !

[lintool]

Hrm, interesting - works for me?

$ python -m unittest tests/test_tokenize_json.py 
Converted 0 docs, writing into file ./test_out_tokenize_json/docs00.json
Converted 0 docs, writing into file ./test_out_tokenize_json/docs01.json
.Converted 0 docs, writing into file out_test_bert_single_file.json
.
----------------------------------------------------------------------
Ran 2 tests in 2.851s

OK

[MXueguang]

what's your sentencepiece/transformers version?

[lintool]

(python36) iMac-Pro:pyserini jimmylin$ pip list | grep transformers
transformers                  4.0.0
(python36) iMac-Pro:pyserini jimmylin$ pip list | grep sentencepiece
sentencepiece                 0.1.95

[MXueguang]

reason is this line should be sorted

pyserini/pyserini/tokenize_json_collection.py

Line 44 in b466646

for i, inf in enumerate(sorted(os.listdir(args.input))):

[MXueguang]

closed by #535