Upgrading from elasticsearch-6.5.4 to 6.6.0 breaks synonym_graph filter

[weberhofer]

Elasticsearch version 6.6.0

Plugins installed: []

JVM version (java -version): cat openjdk version "1.8.0_191"

OS version (uname -a if on a Unix-like system): Linux herkules 4.4.172-86-default #1 SMP Fri Feb 1 06:14:18 UTC 2019 (62e27f4) x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I'm searching using synonym_graph with a list of synonyms. After upddating to 6.6.0 the synonym list seem to be ignored.

Steps to reproduce:
I have defined the following mapping:

{
	"settings": {
		"analysis": {
			"filter": {
				"graph_synonyms": {
					"type": "synonym_graph",
					"synonyms_path": "/var/lib/elasticsearch/synonyms.txt",
					"tokenizer": "standard",
					"ignore_case": true,
					"expand": true
				},
			},
			"analyzer": {
				"search_synonyms_case_insensitive": {
					"type": "custom",
					"tokenizer": "standard",
					"filter": [
						"asciifolding",
						"lowercase",
						"graph_synonyms",
						"standard"
					]
				}
			}
		}
	}
}
...

Sending the following query returns all results including the synonyms in 6.5.4 but only exact matches in 6.6.0:

{
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "fgm",
            "fields": [
              "content"
            ],
            "analyze_wildcard": true,
            "phrase_slop": 1,
            "default_operator": "AND",
            "analyzer": "search_synonyms_case_insensitive"
          }
        }
      ]
    }
  }
}

[elasticmachine]

Pinging @elastic/es-search

[markharwood]

Thanks for raising this.
Can you supply the subset of the synonyms file that corresponds with your example query?

[weberhofer]

Thanks for responding. Here are some of our synonyms:

"Female genital mutilations", "FGM", "Infibulationen", "Female genital mutilation", "Frauenbeschneidungen", "Genital mutilations"
"Strafmilderung", "Commutation"
"Irishwomen", "Irin", "Irishmen", "Ire", "Irishwoman", "Irinnen", "Iren", "Irishman", "Irish"
"Property owner", "Property owners", "Grundstückseigentümer"
"Turk", "Turkish", "Türken", "Turks", "Türkin", "Türkinnen", "Türke"

[jimczi]

@romseygeek can you take a look ? It seems that we build all the PreConfiguredTokenFilter with useFilterForParsingSynonyms set to false so filters are not applied when building the synonym map.

[weberhofer]

Thank you @romseygeek for having a look at this issue. Is there a way to define the filter described in a way that v6.6.0 will accept it?

[romseygeek]

@weberhofer defining a custom asciifolding filter and using that in your analyzer settings should act as a workaround here, as the bug is only for preconfigured tokenfilters.

[weberhofer]

@romseygeek , thanks for your response.
This workaround is fine, but the words from the synonym file seem to be case-sensitive now. I can now find all synonyms for "FGM" (as written in the synonym list), but I don't find anything when I'm searching "fgm".

I'm currently using this definition:

		"filter": {
			"graph_synonyms_3": {
				"type": "synonym_graph",
				"synonyms_path": "/var/lib/elasticsearch/synonyms.txt"
			}
		}

		"filter" : {
			"my_ascii_folding" : {
				"type" : "asciifolding"
			}
		}

		"analyzer": {
			"search_synonyms_case_insensitive_2": {
				"type": "custom",
				"tokenizer": "standard",
				"filter": [
					"my_ascii_folding",
					"graph_synonyms_3",
					"lowercase"
				]
			}
		}
	}