Benchmark against pragmatic segmenter

[Immortalin]

No description provided.

[fnl]

Good point; When I developed the first version, segtok, there were no good benchmark datasets for sentence segmentation around that had sufficient coverage of the tricky cases this library can do. That is, all I found were examples of trivial sentence segmentation problems that virtually any statistical tagger can do well on, too. But if someone has a pointer to a really tough test set with stuff like author abbreviations, enumerations, typos, mathematical and scientific content, and/or social domain text (that might be abusing sentence terminal markers), that would be worth adding. Otherwise, I think the 50+ test cases I have collected as examples of such problems are my current "benchmark": I haven't found a single other library that can do all those cases.

[fnl]

The above being said, what I am currently not interested in or would have time to do is go compare my library manually against another, case-by-case. So if someone wants to fulfill the specific request made by Immortalin here (or you yourself?), please feel free to make that comparison, though. I am sure either library will have its particular strengths.

But that being said, for an unbiased comparison, what would be more important is an impartial sentence segmentation dataset that covers the more tricky cases we find in the wild.

[fnl]

Another interesting tool to compare/benchmark against: https://github.com/nipunsadvilkar/pySBD

Note that pySBD is supposedly based on the Pragmatic Segmenter.

[reepush]

For my use case syntok works just perfect. Thanks @fnl for this project!