Ordering of Arrays and Strings

[springcomp]

See: jmespath/jmespath.js#85

Currently, ordering comparison expressions are only valid for numbers.

However, some popular implementations attempt to convert strings to numbers in order to provide a more intuitive behaviour.
Consider the following example:

{
"status":"GOOD",
"fruits":[
  {
    "name":"Apple",
    "details": {
       "size":500
      }
  },
  {
    "name":"Cherry",
    "details": {
        "size":100
      }
  }]
}

Some users might write the following expressions:

• fruits[? details.size >= `500` ] to filter all fruits whose size is greater than number 500.
• fruits[? details.size >= '500' ] to filter all fruits whose size is greater than string "500".

Caveat: this might conflict with grammar specification that mandates that no implicit conversion happens unless explicitely specified.

[innovate-invent]

I think after witnessing the mess that Javascript has with implicit type conversion, we should add a to_number() function that accepts strings and numbers, and returns a number. Erroring if no number can be derived.

[springcomp]

Do you mean altering the definition of the existing to_number() function?
Currently it returns null instead of raising an error.

[innovate-invent]

hah, woops, forgot that was there: https://jmespath.site/#func-to_number

I think returning null on error was a massive misstep. I have to add null checks everywhere and it is ugly and usually requires two calls to to_number().

[springcomp]

Note when comparing strings, it would be worth to point out that string comparisons in JMESPath will use Unicode codepoints. As a result precomposed characters will not be considered to be equal.

[kit494way]

It is good to use code points for string comparisons, because it is better to use a consistent comparison rule in JMESPath.
If precomposed characters cause problem, it may be useful to implement normalize method like this.
It may be better to normalize before comparison in JMESPath.

[springcomp]

Ordering Arrays and Strings

JEP
Author	Maxime Labelle
Status	draft
Created	23-Jun-2023

Abstract

This JEP standardizes string and array comparisons with respect to the ordering operators.

Motivation

Although JMESPath ordering operators are only defined for numbers, comparing strings is one of the most requested features.
Unsurprisingly, some of the popular programming-language implementations of JMESPath already support
lexicographically comparing strings implicitly.

When referring to a string as a sequence of Unicode code points, it is intuitive to generalize comparison to arrays.

Specification

Ordering Operators

Ordering operators >, >=, <, <= are only valid for arrays, numbers and strings.

Evaluating an element with an expression of any other type with a comparison operator or comparing an element with an expression of a different type altogether will yield a null value, which will result in the element being excluded from the result list.

Comparing strings is performed lexicographically, in order of their corresponding sequences of code points.

Comparing arrays is performed in order of their elements. For instance, using the < less-than operator against two JSON arrays x and y, the expression x < y is evaluated to the result of comparing each element at index i with the expression x[i] < y[i].

For example, given:

search( 'foo[?a<b]', {
  "foo": [
    {"a": "char", "b": "bar"},
    {"a": 2, "b": 1},
    {"a": 1, "b": 2},
  ]})

The three elements in the foo array are evaluated against a < b.

The first element resolves to the comparison “char” < “bar”, which evaluates to false, so the first element is not included in the result.
The second element resolves to 2 < 1, which is false, so the second element is excluded from the result as well.
The third expression resolves to 1 < 2 which evaluates to true, so the third element is included in the result.

The final result of that expression is [{"a": 1, "b": 2}].

Examples

search(foo[?bar<`10`], {"foo": [{"bar": 1}, {"bar": 10}]}) -> [{"bar": 1}]
search([?bar==`10`], [{"bar": 1}, {"bar": 10}]}) -> [{"bar": 10}]
search([?bar==`10`], [{"bar": "1"}, {"bar": "10"}]}) -> []
search([?bar=='10'], [{"bar": "1"}, {"bar": "10"}]}) -> [{"bar": "10"}]

Compliance Tests

New tests will also be added to the filters.json file.

History

Date	Description
26-Jun-2023	Initial draft.
28-Jun-2023	Forbid implicit conversion

[springcomp]

Although we are weary of allowing implicit conversions the specification contains the following example:

• search([?bar==`10`], [{"bar": 1}, {"bar": 10}]}) -> [{"bar": 10}]

Should we special-case comparisons between a string and a number ?

[gibson042]

Although we are weary of allowing implicit conversions the specification contains the following example:

• search([?bar==`10`], [{"bar": 1}, {"bar": 10}]}) -> [{"bar": 10}]

Should we special-case comparisons between a string and a number ?

Unless I'm missing something, that example does not imply any cross-type comparison—10 is numeric in both the expression and the data. But to answer the question, I don't think there should be any such special-casing, which raises awkward considerations around e.g. "1e100" vs. "1e+100" vs. "1E100".

[springcomp]

You are right, I mistakenly read that as comparing with a string. `10` is indeed a number.