DocsClientYamlTestSuiteIT.test {yaml=reference/analysis/analyzers/lang-analyzer/line_1353} fails on 6.x with incorrect offset

[dakrone]

Here's the relevant log:

  1> [2019-02-01T08:26:18,320][ERROR][o.e.s.DocsClientYamlTestSuiteIT] [test] This failing test was generated by documentation starting at reference/analysis/analyzers/lang-analyzer.asciidoc:line_1353. It may include many snippets. See docs/README.asciidoc for an explanation of test generation.
  2> REPRODUCE WITH: ./gradlew :docs:integTestRunner -Dtests.seed=D68E75796C96A039 -Dtests.class=org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT -Dtests.method="test {yaml=reference/analysis/analyzers/lang-analyzer/line_1353}" -Dtests.security.manager=true -Dtests.locale=nn-NO -Dtests.timezone=CST -Dtests.distribution=oss-zip -Dcompiler.java=11 -Druntime.java=8
FAILURE 2.60s | DocsClientYamlTestSuiteIT.test {yaml=reference/analysis/analyzers/lang-analyzer/line_1353} <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Failure at [reference/analysis/analyzers/lang-analyzer:1316]: expected:<{token=⁉, start_offset=2245, end_offset=2246, type=<EMOJI>, position=6560}> but was:<{token=⁉, start_offset=2245, end_offset=2247, type=<EMOJI>, position=6560}>
   > 	at __randomizedtesting.SeedInfo.seed([D68E75796C96A039:5EDA4AA3C26ACDC1]:0)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:406)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:383)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.lang.AssertionError: expected:<{token=⁉, start_offset=2245, end_offset=2246, type=<EMOJI>, position=6560}> but was:<{token=⁉, start_offset=2245, end_offset=2247, type=<EMOJI>, position=6560}>
   > 	at org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT$CompareAnalyzers.execute(DocsClientYamlTestSuiteIT.java:198)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:399)
   > 	... 38 more

This does reproduce on 6.x for me with:

./gradlew :docs:integTestRunner -Dtests.seed=D68E75796C96A039 -Dtests.class=org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT -Dtests.method="test {yaml=reference/analysis/analyzers/lang-analyzer/line_1353}" -Dtests.security.manager=true -Dtests.locale=nn-NO -Dtests.timezone=CST -Dtests.distribution=oss-zip -Dcompiler.java=11 -Druntime.java=8

[elasticmachine]

Pinging @elastic/es-search

[polyfractal]

Took a quick look, and it appears that the "reimplemented" persian analyzer is failing because it doesn't match the actual persian analyzer. It comes from this section:

The `persian` analyzer could be reimplemented as a `custom` analyzer as follows:

PUT /persian_example
{
  "settings": {
    "analysis": {
      "char_filter": {
        "zero_width_spaces": {
            "type":       "mapping",
            "mappings": [ "\\u200C=> "] <1>
        }
      },
      "filter": {
        "persian_stop": {
          "type":       "stop",
          "stopwords":  "_persian_" <2>
        }
      },
      "analyzer": {
        "rebuilt_persian": {
          "tokenizer":     "standard",
          "char_filter": [ "zero_width_spaces" ],
          "filter": [
            "lowercase",
            "decimal_digit",
            "arabic_normalization",
            "persian_normalization",
            "persian_stop"
          ]
        }
      }
    }
  }
}

Unfortunately, that exhausts my debugging skills on this one, since it dives into the composition of the analyzer itself :)

[cbuescher]

This also reproduces on current master (5e71102). I'm suprised the failure only reproduces with the given seed (D68E75796C96A039) but not with a lot of other random seeds. I didn't expect that from a test related to the documentation.

[cbuescher]

I'm suprised the failure only reproduces with the given seed (D68E75796C96A039) but not with a lot of other random seeds. I didn't expect that from a test related to the documentation.

After diving into the DocClientYamlTestSuiteIT a little further, this is explainable by CompareAnalyzers#execute testing the analyzers with a random unicode sequences separated by spaces.

[mayya-sharipova]

The problem with this symbol \u200C in a test string, that SHOULD be substituted with space in the rebuilt Persian analyzer, but it is not.

Correcting this line "mappings": [ "\\u200C=> "] <1> to "mappings": [ "\\u200C=>\\u0020"] <1> in
lang-analyzer solves the problem.
This change explicitly says to substitute ZWNJ with a space.

@cbuescher do you want me to create a PR for this change in lang-analyzer.asciidoc?

[alpar-t]

This still fails on all branches: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+intake/38