Object Manipulation Functions

[innovate-invent]

We need a mechanism for building objects with an unknown number of keys. This has already been extensively discussed and this discussion is to summarise, preparing to create a JEP.

jmespath/jmespath.py#105
https://github.com/jmespath/jmespath.site/issues/17

The proposed functions are:

array[array[string, any]] items(object)
array[array[any, any]] zip([array *argument, [, array $...]])
object from_items(array[array[string, any]])

items() takes an object and returns an array of pairs [key, value] for each entry
zip() behaves exactly like the python zip() function and allows combining an array of keys and array of values into what is required by the next function
from_items() takes an array of pairs [key, value] and builds an object with the respective key->value associations

zip() is a useful operation for other uses and can accept more than two arrays to combine.
The arrays returned by the existing keys() and values() functions can be combined using zip() to reproduce the original object via from_items()

examples:
search({"a":1, "b":2}, items(@)) -> [["a", 1], ["b", 2]]
search([["a", 1], ["b", 2]], from_items(@)) -> {"a":1, "b":2}
search({"a":[1, 2, 3], "b":[4,5,6], "c":[7,8,9]}, zip(a, b, c)) -> [[1,4,7], [2,5,8], [3,6,9]]

an alternative to items() could be search({"a":1, "b":2}, let({"k":keys(@), "v":values(@)}, &map(&[k[@], v[@]], range(length(@))))) where range() is the same as the python function by the same name.

an alternative to zip() could be search({"a":[1, 2, 3], "b":[4,5,6], "c":[7,8,9]}, let({a, b, c}, &map(&[a[@], b[@], c[@]], range(len(a)))))
This pattern allows specifying which array determines the final length, while the index accessors return null for any list shorter than the specified one.

an alternative to from_items() could be search([["a", 1], ["b", 2]], merge(map(&{@[0] : @[1]}, @)))
This requires expanding the grammar to allow expressions in place of the key. merge() also needs to allow accepting a single array argument and operating over that as if it were its argument list.

I think with these alternatives, we need to discuss how strictly we want to adhere to the "one way of doing something" guideline. Where do we draw the line between a wiki of reusable patterns and adding functions to simplify?

[innovate-invent]

The more I think about this, the more I am worried we are adding functions that should actually be syntax.

This is most apparent when considering concat() vs merge(), why does concat() exist as syntax and merge() doesn't?
concat() is implemented as the flatten operation: [[1], [2]][] -> [1,2]
should this be extended to include objects?: [{"a":1}, {"a":2, "b":1}][] -> {"a":2, "b":1}
we could add syntax such that it would be: [{"a":1}, {"a":2, "b":1}]{} -> {"a":2, "b":1}

alternatively we could look at adding a spread operator to multiselect list and hash:
[[1].., [2]..] -> [1,2]
{{"a":1}.., {"a":2, "b":1}..} -> {"a":2, "b":1}

We also have two different ways of applying the map() operation: map(&to_string(@), k) vs k[*].to_string(@)

If we go this route then the alternatives would be:
items: let({"k":keys(@), "v":values(@)}, range(length(@))[*].[k[@], v[@]]) or let({a: @}, keys(@)[*].[@, a[@]])
zip: let({a, b, c}, range(len(a))[*].[a[@], b[@], c[@]])
from_items: [*].{[0] : [1]} | []

or with the reduce operation
from_items: [&].[acc, {cur[0]:cur[1]}]{}

or with the spread operator:
from_items: [&].{acc.., cur[0]: cur[1]}

or with the set union operator:
from_items: [&].(acc + {cur[0]: cur[1]})

[innovate-invent]

The from_items examples would require modifying the keyval-expr = identifier ":" expression syntax to be keyval-expr = expression ":" expression, or possibly keyval-expr = (paren-expression / identifier) ":" expression for backwards compatibility. Allowing dynamic keys would be useful in other situations.

[springcomp]

Object Manipulation Functions

JEP	13
Author	Jonathan Stewmon
Created	21-March-2016
SemVer	MINOR
Status	Draft
[Discussion #47]	#47

Abstract

As a JSON querying language, JMESPath has long needed some functions to manipulate JSON objects.

This JEP introduces built-in functions to extract a list of key/value pairs from a JSON object and conversely converting an array of key/value pairs to a JSON object.

Specification

items

array[array[any]] items(object $obj)

Returns a an array of key value pairs for the provided object $obj. Each pair is a 2-item array with the first item being the key and the second item being the value. This function is the inverse of the from_items() function.

Note that because JSON hashes are inheritently unordered, the key value pairs of the provided object $obj are inheritently unordered. Implementations are not required to return values in any specific order.

For example, given the input:

{"a": "first", "b": "second", "c": "third"}

The expression items(@) could have any of these return values:

• [["a", "first"], ["b", "second"], ["c", "third"]]
• [["a", "first"], ["c", "third"], ["b", "second"]]
• [["b", "second"], ["a", "first"], ["c", "third"]]
• [["b", "second"], ["c", "third"], ["a", "first"]]
• [["c", "third"], ["a", "first"], ["b", "second"]]
• [["c", "third"], ["b", "second"], ["a", "first"]]

If you would like a specific order, consider using the sort_by function.

Examples

Given	Expression	Result
{"a": "first", "b": "second"}	items(@)	[["b", "second"], ["a", "first"]]
{"z": "last", "b": "second"}	sort_by(items(@), &[0])	[["b", "second"], ["z", "last"]]
{"z": "last", "b": "second"}	sort_by(items(@), &[1])	[["z", "last"], ["b", "second"]]

from_items

object from_items(array[array[any]] $arg)

Returns an object from the provided array of key value pairs. This function is the inverse of the items().

Examples

Given	Expression	Result
[["one", 1], ["two", 2]]	from_items(@)	{"one": 1, "two": 2}
[["one", 1], ["two", 2], ["one", 3]]	from_items(@)	{"one": 3, "two": 2}

zip

array[array[any]] zip([array[any] $arg, [, array[any] $...]])

Accepts one or more arrays as arguments and returns an array of arrays in which the i-th array contains the i-th element from each of the argument arrays. The returned array is truncated to the length of the shortest argument array.

Examples

Expression	Result
zip(`["a", "b"]`, `[1, 2]`)	[["a", 1], ["b", 2]]
zip(`["a", "b", "c"]`, `[1, 2]`)	[["a", 1], ["b", 2]]

Compliance tests

A new objects.json file will be added to the compliance test suite.

History

N/A

[springcomp]

@jstewmon I took the liberty to use your pull request and turn this into a JEP. You may want to share your feedback when discussing this JEP.

@gibson042 words to the effect that implementations are required to preserve source order would not bother me so much. Would that be OK for you?

[jstewmon]

👋 Hi, thanks for thinking of me. I'm not currently using jmespath in a project, so I don't feel inclined to argue any points strongly, but I'll leave a few remarks on how I see things. 😄

I think everything in the parent post looks good.

IMO, the ordering behavior of items should be consistent with keys and values:

jmespath.spec/functions/keys.yml

Lines 13 to 17 in 4a8c323

	Returns an array containing the keys of the provided object.
	Note that because JSON hashes are inheritently unordered, the
	keys associated with the provided object `obj` are inheritently
	unordered. Implementations are not required to return keys in
	any specific order.

jmespath.spec/functions/values.yml

Lines 13 to 44 in 4a8c323

	Returns the values of the provided object.
	Note that because JSON hashes are inheritently unordered, the
	values associated with the provided object `obj` are inheritently
	unordered. Implementations are not required to return values in
	any specific order. For example, given the input:
	```
	{"a": "first", "b": "second", "c": "third"}
	```
	The expression `values(@)` could have any of these return values:
	* `["first", "second", "third"]`
	* `["first", "third", "second"]`
	* `["second", "first", "third"]`
	* `["second", "third", "first"]`
	* `["third", "first", "second"]`
	* `["third", "second", "first"]`
	If you would like a specific order, consider using the
	`sort` or `sort_by` functions.

I think the originally proposed description (below) is consistent with the description of keys:

Note that because JSON hashes are inheritently unordered, the key value pairs of the provided object $obj are inheritently unordered. Implementations are not required to return values in any specific order.

I don't think it's appropriate to debate the ordering semantics of items in this proposal. If there's cause to debate the ordering semantics of all the functions that return an array from an object, make a separate proposal and debate it there.

[gibson042]

IMO, the ordering behavior of items should be consistent with keys and values:

I don't think it's appropriate to debate the ordering semantics of items in this proposal. If there's cause to debate the ordering semantics of all the functions that return an array from an object, make a separate proposal and debate it there.

Excellent point. Spun off to #94 .

[springcomp]

Good. Thanks a lot for chiming in @jstewmon . Your feedback was precious.

[springcomp]

For information, those functions are demonstrated here.

[springcomp]

Should zip() be included in this proposal?

[innovate-invent]

haha, I was literally typing a correction as you posted this.

it isn't the reduce operation that competes, it is the arithmetic-operators.

zip() is equivalent to the cartesian product [a,b] * [1,2] -> [[a,1],[b,2]] or even [a,b] * [1,2] * [c,d] -> [[a,1,c],[b,2,d]]

[springcomp]

Ah I see. Maybe we need to settle on basic arithmetic operations first, before moving on to vectors and matrices 😊
I think zip() does not necessarily belongs to this JEP.

[jstewmon]

zip() is equivalent to the cartesian product [a,b] * [1,2] -> [[a,1],[b,2]] or even [a,b] * [1,2] * [c,d] -> [[a,1,c],[b,2,d]]

This is not true.

The cartesian product has a length equal to the product of the input lengths. The cartesian product of [a, b] * [1, 2] = [[a, 1], [a, 2], [b, 1], [b, 2]].

The zip result has length equal to the min length of its inputs. The zip function can be though of as correlating the items from each input list by their ordinal position. zip([a, b], [1, 2]) = [[a, 1], [b, 2]].

zip is commonly used to produce an argument for from_items. Consider tabular data, like a database query result set or CSV - the field labels are commonly defined once (e.g. the first row, or a separate data structure) in a sequence with ordinal positions corresponding to the position of their values in the data rows. If you want to produce an object/map from each data row, then you can call from_items(zip(fields, values)) where values is a data row.

[innovate-invent]

sorry, you are right!
I was struggling to find my example of an alternative to zip() that I had previously posted, saw my proposal of the product of objects and just went with it without looking too close.

The actual competing syntax would be search({x: [a,b], y:[1,2]}, let({y: y}}, x[*].[@, y[$]]) which is using let(), dynamic subscription, and $

I do want to explore a operator syntax alternative to zip though

[jstewmon]

Personally, I find that dynamic subscription syntax inscrutable. 😅 Zip may not be ubiquitous, but it is familiar from the Python and Ruby standard libraries, lodash for JavaScript, etc.

Anyway, like I said in my other comment, I'm not really using jmespath anymore, so I don't have a stake in the outcome.