Undocumented behaviour / Bug: ElasticSearch Painless - (Vector) Functions in (for) loops

[coreation]

Elasticsearch version (bin/elasticsearch --version): 7.11

Plugins installed: none outside the default ones coming with 7.11

JVM version (java -version): Java8

OS version (uname -a if on a Unix-like system): OS X

Description of the problem including expected versus actual behavior:

I'm currently experiencing something weird in the following script:

def scores =[]; def xt = params.filterVectors; for (int i=0; i< xt.size();i++) { scores.add(cosineSimilarity(xt[i], doc['media.full_body_dense_vector'].asList()));} throw new Exception(scores.toString());

The result that is added to the array, is always the cosineSimilarity outcome of the first iteration, however if you log "xt[i]", it is looping through the passed parameters of the script. It seems however that if a (vector) function is passed in a for loop, it's always the same outcome as in the first iteration. In other words it doesn't seem like you can dynamically re-use the function inside a for loop. (?)

**I have no issue with providing more detail If someone could tell me that this is a bug, or just undocumented behaviour. By undocumented I mean that there's documentation on using for loops, and documentation on a variety of vector functions (much appreciated btw!!), but there's nothing that states you can't use them inside a loop.

Steps to reproduce:

1. mapping

"mappings": {
  "properties": {
    "title": {
      "type": "text"
    },
    "title_vector": {
      "type": "dense_vector",
      "dims": 512
    }
    "tags": {
      "type": "keyword"
    }
  }
}

2. script / query

    {
    	"query": {
    		"function_score": {
    			"boost_mode": "replace",
    			"query": {
    				"bool": {
    					"filter": [{
    						"exists": {
    							"field": "title_vector"
    						}
    					}]
    				}
    			},
    			"functions": [{
    				"script_score": {
    					"script": {
    						"source": "def scores =[]; def xt = params.filterVectors; for (int i=0; i< xt.size();i++) { scores.add(cosineSimilarity(xt[i], doc['title_vector']));} throw new Exception(scores.toString());",
    						"params": {
    							"filterVectors": [
    								[1.0, 2.0, 3.0],
    								[0.1, 0.4, 0.5]
    							]
    						},
    						"lang": "painless"
    					}
    				}
    			}]
    		}
    	},
    	"size": 500,
    	"from": 0,
    	"track_scores": true
    }

Provide logs (if relevant):

No logs are produced, just the exception which prints 2 times the same calculation outcome.

Link to relevant StackOverflow ticket

https://stackoverflow.com/questions/66639279/elasticsearch-painless-using-vector-functions-in-for-loops-bug/66662131#66662131

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[jdconrad]

@mayya-sharipova Would you please take a look at this? I don't know what the expected behavior for cosineSimilarity is well enough to determine if this is indeed a bug or not.

It looks like all the parameters are intentionally "locked" at the first invocation as they are all part of the constructor of a Painless class binding. I see in the history there was at one point a mutable parameter passed into the method of the class binding that was removed in #48604.

[jdconrad]

@mayya-sharipova @rjernst looked through the code with me and we both think it should work as-is. I will take some time to try to reproduce and see if we can figure out what isn't getting updated as we think it should be.

[coreation]

@jdconrad maybe @mayya-sharipova 's reply helps, she provided a reproducable flow on this comment.

[jdconrad]

I have found the issue here. We have certain methods that we whitelist that fall under the category of class binding. A class binding in Painless is a method that we internally create a new instance of a class for and pass specific arguments that we consider read-only to a constructor for caching and other mutable arguments to a method that is invoked each time. We do this for performance reasons as this set of methods would otherwise be prohibitive to use repetitively if a large number of documents match a search.

cosineSimilarity falls under the class binding category and both arguments are passed to the constructor and cached with some work done on them. When the method is invoked a second time the originally passed in arguments are used as opposed to mutable ones.

I see three possible options that I would like to discuss as alternatives to what exists now:

1. We add/improve documentation to describe the situation here and leave it as-is, but that means that these methods cannot be used in a loop.
2. We modify the method to process the vectors each time, but this means we lose the original purpose of a class binding to have good performance.
3. We provide an alternative method that's the "slow" path but does accept new data each time as well as the original "fast" path method.

Given our general goal to make Painless as user-friendly as possible, 3 is probably the best choice, but I'd like to hear thoughts from other members of the team on this. @rjernst @stu-elastic @mayya-sharipova @jtibshirani

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jzzfs]

@jdconrad my 2 cents if I may: I don't think it's only limited to cosineSimilarity & other vector functions but essentially to all static Painless methods that exhibit this class binding behavior. I debugged l2norm in this SO issue, for instance:

"script": {
  "source": """
      def field = doc['media.full_body_dense_vector'];
      
      def hardcodedVectors = [ [1,2,3], [0.1,0.4,0.5] ];
      
      def noLoopDistances = [
        l2norm(hardcodedVectors[0], field),
        l2norm(hardcodedVectors[1], field)
      ];
      
      def hardcodedDistances = [];
      for (vector in hardcodedVectors) {
        double euDistance = l2norm(vector, field);
        hardcodedDistances.add(euDistance);
      }
      
      def parametrizedDistances = [];
      for (vector in params.filterVectors) {
        double euDistance = l2norm(vector, field);
        parametrizedDistances.add(euDistance);
      }
      
      def comparisonMap = [
        "no-loop": noLoopDistances,
        "hardcoded": hardcodedDistances,
        "parametrized": parametrizedDistances
      ];
      
      Debug.explain(comparisonMap);
     """,
  "params": {
    "filterVectors": [ [1,2,3], [0.1,0.4,0.5] ]
  },
  "lang": "painless"
}

yielding

{
  "no-loop":[             
    8.558621384311845,    // <-- the only time two different l2norm calls behave correctly
    11.071133967619906
  ],
  "parametrized":[
    8.558621384311845,
    8.558621384311845
  ],
  "hardcoded":[
    8.558621384311845,
    8.558621384311845
  ]
}

I also tried to call l2norm in a separate function def but then the compiler complains about the this context.

When I try to pass the top-level l2norm to a separate function as an argument, the compiler doesn't recognize the l2norm keyword (unless it was of course called with arguments that it expects).