Fixup/pr/5881 - Add source filtering

[spalger]

In order to apply final fixes before merging, moving this from #5881 so we can iterate faster.

---

• fetch field if:
  • explicitly selected in discover
  • show in a table
  • used in an aggregate

---

close #4366

I am not sure if this PR is a complete solution for the issue, but I believe it is a part of it.

This PR adds a configuration on the index pattern level that allows to define which field to exclude using the source filtering API. There is a new tab in which you can set the JSON to specify what to exclude.
The fields that are actually retrieved are indicated on the Fields tab as an additional column: if it is retrieved, the row is checked.

To test it:

• Create a small index:

curl -XPUT 'http://localhost:9200/country/Country/1' -d '
{
  "name": "Italy",
  "description": "pasta"
}'

• Add the country index and go to the Retrieved Fields tab. Exclude the description field from being retrieved. Add the following object to do so:

{
 "exclude": "description"
}

• The description is now unchecked, meaning it is not retrieved.

• Create a saved search for the countries and then add a search visualisation to a dashboard. You will see that the field description does not appear.

---

What is not provided in this PR but could be necessary: a way to display an excluded field for a document. Do you see this mandatory, or is the proposed solution good enough ?

[jbudz]

Looking good!

1. As brought up outside, can we add a note saying this isn't a security feature?
2. If I add a column in discover for example, and then exclude it, all the values show up as nulls. Thoughts on this potentially being misleading?
   

[spalger]

Talked with @rashidkpc about this and we agree that this is fine for now. If someone configures the index pattern to exclude that field then that's what they get. If it turns out that isn't sufficient we can cross that bridge.

[scampi]

I tried the branch, and there is a little left-over from the previous PR.

• As pointed out by @jbudz , selecting a column in Discover does not retrieve values for it when it is excluded.
  We can then update the piece of code in https://github.com/elastic/kibana/pull/5981/files#diff-274005236556ec7aa3ebc5475ca54f5bR127 so that getSourceFiltering does not take columns as argument.
• the test at https://github.com/elastic/kibana/pull/5981/files#diff-cda2901f9787a94276f3f175167a3311R74 can be removed.

@jbudz @spalger thanks for taking care of this. If you need me to do something, please ping me.

[jccq]

Hi guys i would reccomend an option (To go with this new feature) to show the 'hidden" field when browsing the specific record e.g. in the result table.

one should see something like "there are 3 fileds that have not been loaded by default e.g. "full description" "comment lists" etc.. and then a button that retrieves it for the operator that is interested in actually looking at them..

i feel this extra feature is really needed so not to "lose sight of data" (conceptually) ever. What do you think?

[spalger]

I'm inclined to agree @jccq. Unfortunately it's much more difficult/complex to implement a solution where discover or other views have control over what fields are being filtered.

[w33ble]

Much better!

[w33ble]

With the latest commits, this LGTM. Passing to @Bargs for final confirmation, since he's looked at this before.

[Bargs]

Should this work on meta fields? I tried excluding _id, and it matched, but it didn't actually get exluded from discover.

[Bargs]

Typos:

[Bargs]

Given that this is a brand new feature, it probably deserves some doc updates now that we're responsible for documentation. Especially because even I don't understand where all this filtering is expected to take place. The original PR description mentions filtering in the dashboard, but that no longer appears to happen.

Are there any places besides discover where this filtering is currently expected to take place?

[Bargs]

How difficult would it be to hide the filter bar from this tab, since it doesn't do anything?

[Bargs]

It matches on scripted fields but doesn't filter them:

[scampi]

fixes will be done in #7402
This can be closed