[Security Solution] [Timeline] Bugfix to include unmapped fields in the timeline event details JSON

[stephmilovic]

Summary

Resolves the following issues in timeline related to the ES Fields API change:

#91424
#90355
#90222
#91426
#90808

Does NOT resolve: #89784 Dragging a nested field to timeline does not generate the correct query

• To be fixed in a follow up PR.
• At least adding it with the + button works, just dragging creates bad query
• + Filter also needs display to be fixed, but is querying correctly

Click here to see the nested filter issue

Updates the timeline event details query dsl to include fields: ['*'] and _source: true in order to ensure we are getting all fields, even unmapped. Array values will be stringified

Also alphabetizes the JSON view to make it nice and easy to read for Prince @spong 👑

I also fixed a bug I noticed that made stringified object arrays into draggables, yikes:

I changed this to render as an undraggable string:

FWIW, the actual values were already being parsed correctly into draggable below:

To test:

post the below data to your siem-signals-default index. on the detections table page, search for signal.rule.name": "GRRRRRowl", Ensure the threat.indicator and source.geo.location appear appropriately as draggables and strings where the full object is listed.

POST .siem-signals-default/_doc
{
  "@timestamp": "2021-02-26T00:41:06.527Z",
  "signal.status": "open",
  "signal.rule.name": "GRRRRRowl",
  "threat.indicator": [
    {
      "provider": "yourself",
      "type": "custom",
      "first_seen": [
        "2021-02-22T17:29:25.195Z"
      ],
      "matched": {
        "atomic": "matched_atomic",
        "field": "matched_field"
      }
    },
    {
      "provider": "other_you",
      "type": "custom",
      "first_seen": "2021-02-22T17:29:25.195Z",
      "matched": {
        "atomic": "matched_atomic_2",
        "field": "matched_field_2"
      }
    }
  ],
  "source": {
    "geo": {
      "location": [
        {
          "lat": 39.5884,
          "lon": -105.0776
        }
      ]
    }
  }
}

[stephmilovic]

I "de-alphabetized" this mock data to ensure that our function puts it in alphabetical order

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[stephmilovic]

adding this data (result on 227) tests toObjectArrayOfStrings

[michaelolo24]

Just curious, were the ${eventId}-${field.id}-${value} not unique? I ask because we ran into an issue on the resolver when we introduced indices into keys because reacts resolution on re-renders would be jumbled if the data changed

[stephmilovic]

it would throw the error on the nested field

[stephmilovic]

i wonder if this was before or after i filtered for unique values though, ill take a second look

[michaelolo24]

👍🏾 nice cleanup

[michaelolo24]

I think the await goes outside the expect? https://jestjs.io/docs/en/tutorial-async#asyncawait

[stephmilovic]

I think thats for with .resolves

In the first example:

it('works with async/await', async () => {
  expect.assertions(1);
  const data = await user.getUserName(4);
  expect(data).toEqual('Mark');
});

so mine is like this, but instead of using a variable data i put the await function call directly in the () because it will run what is inside the () before evaluating the expect

[rylnd]

Not going to Approve because I reviewed behavior and not code here, but things look great on my end! Filtering works as documented in the description

Display of multiple nested documents:

[rylnd]

@stephmilovic one other thing I noticed: sorting on those nested fields causes the query to fail:

I don't think that should block this effort, but I wanted to make sure we're aware of it moving forward. The semantics of sorting on a nested field are unclear in the existing UI; would it be possible to disable sorting on those fields for now?

[michaelolo24]

Tested and it looks good! I was able to load the sample data and saw the nested objects properly formatted. Only had a few code comments, but nothing breaking. Feel free to address in a separate PR. Thanks for doing this!

[kqualters-elastic]

pulled and tested locally, does fix the issues. might make sense to add tests for isObjectArray otherwise lgtm pending @michaelolo24 's comment about await

[stephmilovic]

@stephmilovic one other thing I noticed: sorting on those nested fields causes the query to fail:

I don't think that should block this effort, but I wanted to make sure we're aware of it moving forward. The semantics of sorting on a nested field are unclear in the existing UI; would it be possible to disable sorting on those fields for now?

good catch should be able to disable sorting

[stephmilovic]

disable sorting on nested fields, not working.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 08dce77
• Storybooks Preview
• Documentation Changes

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	7.8MB	7.8MB	+1014.0B
triggersActionsUi	1.6MB	1.5MB	-23.9KB
total			-22.9KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
triggersActionsUi	104.0KB	104.1KB	+82.0B

Unknown metric groups

async chunk count

id	before	after	diff
triggersActionsUi	41	42	+1

History

• 💔 Build #110128 failed c14794b
• 💚 Build #110017 succeeded f72b23f
• 💔 Build #109736 failed 3dae543
• 💔 Build #109670 failed 46513ac
• 💔 Build #109446 failed 75a3a4f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream