Major literal optimization refactoring.

[BurntSushi]

The principle change in this commit is a complete rewrite of how
literals are detected from a regular expression. In particular, we now
traverse the abstract syntax to discover literals instead of the
compiled byte code. This permits more tuneable control over which and
how many literals are extracted, and is now exposed in the
regex-syntax crate so that others can benefit from it.

Other changes in this commit:

• The Boyer-Moore algorithm was rewritten to use my own concoction based
  on frequency analysis. We end up regressing on a couple benchmarks
  slightly because of this, but gain in some others and in general should
  be faster in a broader number of cases. (Principally because we try to
  run memchr on the rarest byte in a literal.) This should also greatly
  improve handling of non-Western text.
• A "reverse suffix" literal optimization was added. That is, if suffix
  literals exist but no prefix literals exist, then we can quickly scan
  for suffix matches and then run the DFA in reverse to find matches.
  (I'm not aware of any other regex engine that does this.)
• The mutex-based pool has been replaced with a spinlock-based pool
  (from the new mempool crate). This reduces some amount of constant
  overhead and improves several benchmarks that either search short
  haystacks or find many matches in long haystacks.
• Search parameters have been refactored.
• RegexSet can now contain 0 or more regular expressions (previously, it
  could only contain 2 or more). The InvalidSet error variant is now
  deprecated.
• A bug in computing start states was fixed. Namely, the DFA assumed the
  start states was always the first instruction, which is trivially
  wrong for an expression like ^☃$. This bug persisted because it
  typically occurred when a literal optimization would otherwise run.
• A new CLI tool, regex-debug, has been added as a non-published
  sub-crate. The CLI tool can answer various facts about regular
  expressions, such as printing its AST, its compiled byte code or its
  detected literals.
• A new public method on Regex, shortest_match, was added for
  providing is_match performance with some location information.

Closes #96, #188, #189

[BurntSushi]

Benchmark diff, removed <5% noise:

name                                    rust.master ns/iter   rust.lits ns/iter       diff ns/iter   diff %
misc::anchored_literal_long_match       76 (5,131 MB/s)       59 (6,610 MB/s)                  -17  -22.37%
misc::anchored_literal_long_non_match   55 (7,090 MB/s)       15 (26,000 MB/s)                 -40  -72.73%
misc::anchored_literal_short_match      76 (342 MB/s)         54 (481 MB/s)                    -22  -28.95%
misc::anchored_literal_short_non_match  55 (472 MB/s)         23 (1,130 MB/s)                  -32  -58.18%
misc::easy0_1K                          241 (4,360 MB/s)      254 (4,137 MB/s)                  13    5.39%
misc::easy1_1K                          204 (5,117 MB/s)      269 (3,881 MB/s)                  65   31.86%
misc::easy1_32                          183 (284 MB/s)        170 (305 MB/s)                   -13   -7.10%
misc::easy1_32K                         973 (33,697 MB/s)     1,041 (31,496 MB/s)               68    6.99%
misc::hard_1K                           4,855 (216 MB/s)      242 (4,342 MB/s)              -4,613  -95.02%
misc::hard_1MB                          4,775,026 (219 MB/s)  39,144 (26,788 MB/s)      -4,735,882  -99.18%
misc::hard_32                           334 (176 MB/s)        186 (317 MB/s)                  -148  -44.31%
misc::hard_32K                          149,266 (219 MB/s)    1,009 (32,502 MB/s)         -148,257  -99.32%
misc::literal                           19 (2,684 MB/s)       21 (2,428 MB/s)                    2   10.53%
misc::medium_1K                         578 (1,820 MB/s)      248 (4,241 MB/s)                -330  -57.09%
misc::medium_1MB                        344,660 (3,042 MB/s)  39,288 (26,690 MB/s)        -305,372  -88.60%
misc::medium_32                         243 (246 MB/s)        192 (312 MB/s)                   -51  -20.99%
misc::medium_32K                        10,995 (2,982 MB/s)   1,022 (32,090 MB/s)           -9,973  -90.70%
misc::no_exponential                    536 (186 MB/s)        571 (175 MB/s)                    35    6.53%
misc::not_literal                       301 (169 MB/s)        270 (188 MB/s)                   -31  -10.30%
misc::one_pass_long_prefix              182 (142 MB/s)        204 (127 MB/s)                    22   12.09%
misc::one_pass_long_prefix_not          184 (141 MB/s)        160 (162 MB/s)                   -24  -13.04%
misc::one_pass_short                    142 (119 MB/s)        124 (137 MB/s)                   -18  -12.68%
misc::one_pass_short_not                145 (117 MB/s)        119 (142 MB/s)                   -26  -17.93%
misc::reallyhard_32                     332 (177 MB/s)        310 (190 MB/s)                   -22   -6.63%
misc::replace_all                       149                   159                               10    6.71%
sherlock::before_holmes                 2,753,975 (216 MB/s)  118,858 (5,005 MB/s)      -2,635,117  -95.68%
sherlock::everything_greedy             7,961,762 (74 MB/s)   7,339,669 (81 MB/s)         -622,093   -7.81%
sherlock::everything_greedy_nl          5,485,444 (108 MB/s)  5,783,734 (102 MB/s)         298,290    5.44%
sherlock::ing_suffix                    3,185,292 (186 MB/s)  645,587 (921 MB/s)        -2,539,705  -79.73%
sherlock::ing_suffix_limited_space      3,515,374 (169 MB/s)  2,947,184 (201 MB/s)        -568,190  -16.16%
sherlock::letters                       60,357,656 (9 MB/s)   35,502,155 (16 MB/s)     -24,855,501  -41.18%
sherlock::letters_lower                 58,671,286 (10 MB/s)  33,555,537 (17 MB/s)     -25,115,749  -42.81%
sherlock::letters_upper                 4,603,948 (129 MB/s)  3,780,972 (157 MB/s)        -822,976  -17.88%
sherlock::name_alt3_nocase              1,467,278 (405 MB/s)  1,379,928 (431 MB/s)         -87,350   -5.95%
sherlock::name_alt4                     290,928 (2,044 MB/s)  260,026 (2,287 MB/s)         -30,902  -10.62%
sherlock::name_alt4_nocase              1,428,730 (416 MB/s)  1,351,727 (440 MB/s)         -77,003   -5.39%
sherlock::name_alt5_nocase              1,426,142 (417 MB/s)  1,351,901 (440 MB/s)         -74,241   -5.21%
sherlock::name_holmes                   50,481 (11,785 MB/s)  54,785 (10,859 MB/s)           4,304    8.53%
sherlock::name_sherlock                 34,667 (17,161 MB/s)  78,590 (7,570 MB/s)           43,923  126.70%
sherlock::name_sherlock_holmes          34,375 (17,307 MB/s)  41,168 (14,451 MB/s)           6,793   19.76%
sherlock::name_sherlock_holmes_nocase   1,216,050 (489 MB/s)  1,147,994 (518 MB/s)         -68,056   -5.60%
sherlock::name_sherlock_nocase          1,216,131 (489 MB/s)  1,146,695 (518 MB/s)         -69,436   -5.71%
sherlock::name_whitespace               60,939 (9,762 MB/s)   98,650 (6,030 MB/s)           37,711   61.88%
sherlock::no_match_common               558,796 (1,064 MB/s)  26,581 (22,381 MB/s)        -532,215  -95.24%
sherlock::no_match_really_common        574,266 (1,035 MB/s)  411,561 (1,445 MB/s)        -162,705  -28.33%
sherlock::no_match_uncommon             24,136 (24,649 MB/s)  26,542 (22,414 MB/s)           2,406    9.97%
sherlock::quotes                        985,020 (603 MB/s)    922,841 (644 MB/s)           -62,179   -6.31%
sherlock::the_lower                     771,175 (771 MB/s)    730,353 (814 MB/s)           -40,822   -5.29%
sherlock::the_upper                     53,009 (11,223 MB/s)  59,379 (10,019 MB/s)           6,370   12.02%
sherlock::the_whitespace                2,005,052 (296 MB/s)  1,603,387 (371 MB/s)        -401,665  -20.03%
sherlock::words                         19,573,776 (30 MB/s)  13,545,584 (43 MB/s)      -6,028,192  -30.80%

[BurntSushi]

Looks like the nightly build is failing due to: rust-lang/rust#32532

[alexcrichton]

Holy cow @BurntSushi, you're a machine! I can't really find a downside here, so r=me whenever you're ready to go :)

[aturon]

A new CLI tool, regex-debug, has been added as a non-published sub-crate

I love how this is just, you know, casually tossed in there. Amazing stuff, @BurntSushi! 🍣

[BurntSushi]

Thanks! :)

[BurntSushi]

This is on crates.io in 0.1.59.