Queryable object fields

[clintongormley]

Often we have large object fields with many sub-fields, only a few of which are needed for aggregations, sorting, or highlighting. Today, we create fields for all sub-fields, but we could greatly reduce the number of required fields if we make object fields queryable.

We would need a specialiased analyzer which can accept JSON and transform an object like:

{
    "status": "active",
    "age": 25,
    "city": "New York"
}

into the following terms:

"status:active", "age:25", "city:new", "city:york"

Then you could search for active statuses with:

{
  "match": {
    "my_object": "status:active"
  }
}

or

{
  "query_string": {
    "query": "my_object:\"status:active\""
  }
}

We could possibly even support searching for "New York" vs "New City of York" with:

{
  "match_phrase": {
    "my_object": "New York"
  }
}

which would be rewritten as my_object:"city:new city:york"

If we wanted to be able to aggregate on the age field, the object field could be mapped as:

{
  "my_object": {
    "type": "object",
    "index": true,
    "dynamic": false,
    "properties": {
      "age": {
        "type": "integer"
      }
    }
  }
}

With this mapping, only the my_object.age sub-field would have its own Lucene field (or Elasticsearch field) and the rest of the object would be queryable via the my_object field.

This could even be made to work on the whole document by allowing the _source field to be configurable.

[clintongormley]

We could also index terms with and without the field prefix, (eg status:active, active) to make it work more like a normal field

[nik9000]

I wonder if you could make this more transparent so the normal thing works:

PUT idx/_mapping
{
  "doc": {
    "properties": {
      "my_object": {
        "type": "keyvalue",
        "value_type": "keyword"
      }
    }
  }
}

POST idx/_search
{
  "match": {
    "my_object.status": "active"
  }
}

The prefixing would be an implementation detail of keyvalue. I'd be inclined to only support keyword and text style fields in there. That'd let you tell someone "stick whatever you want in this field and we'll index it and you won't pay a cost every time you introduce a new field". I feel like giving numbers their own lucene field makes this less foolproof. I think for something like this more foolproof is better than more featureful.

[s1monw]

I wonder if you could make this more transparent so the normal thing works:

I think we should not make it complicated restrictive. It's very simple, we decide how to index and make things work without any special type. you just mark the mapping as "dynamic": false and "indexed" : true. I don't necessarily understand what is trappy here, we don't support these parameters there yet.

The prefixing would be an implementation detail of keyvalue. I'd be inclined to only support keyword and text style fields in there. That'd let you tell someone "stick whatever you want in this field and we'll index it and you won't pay a cost every time you introduce a new field".

I don't get what you mean here, I assume you want to barf on numerics? I think we should just use make text out of it and be done with it. We can add some position inc gaps to make phrases work but from this perspective it's really just one big pile of text.

I feel like giving numbers their own lucene field makes this less foolproof. I think for something like this more foolproof is better than more featureful.

you know they can't use any kind of numeric special things here since we won't support aggregations etc. Let's keep it simple and don't have many exceptions. That's what I'd like though.
I really don't want to get into the number / date / lat,long detection game when it's coming as a string and next time as a number it's going to be a bad user experience and this is support to fix stuff that is changing on a regular basis ie. twitter data.

[nik9000]

I think we should just use make text out of it and be done with it.

That is what I mean.

The way I read the original proposal I thought that we would automatically give numerics their own Lucene field somehow and I didn't like that. I agree that we shouldn't do numerics.

In Clint's example you can pull out numeric fields. I wonder if we can exclude those fields from the indexed object fields.

Could we make queries to the indexed objects look "normal" like I had in my example?

I think it'd be nice to have an example of configuring the field type of the index - whether the strings are analyzed like text or keyword and how you'd set up multifields. And setting up the analyzer/normalizer/etc.

[s1monw]

The way I read the original proposal I thought that we would automatically give numerics their own Lucene field somehow and I didn't like that. I agree that we shouldn't do numerics.
In Clint's example you can pull out numeric fields. I wonder if we can exclude those fields from the indexed object fields.

we are on the same page here. let not be smart but simple

[clintongormley]

Discussed in FixItFriday. We're going to start simple and see what feedback we get. We will index each term alone plus with a path prefix (eg path\0term), so users can query my_object:term but we could also automatically convert my_object.some.path:term to a query for my_object:some.path\0term.