[RFC] Create ELF Fieldset - Stage 2 Proposal

[peasead]

• Have you signed the contributor license agreement? Yes
• Have you followed the contributor guidelines? Yes
• For proposing substantial changes or additions to the schema, have you reviewed the RFC process? Yes
• If submitting code/script changes, have you verified all tests pass locally using make test? Yes
• If submitting schema/fields updates, have you generated new artifacts by running make and committed those changes? Yes
• Is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed. Yes
• Have you added an entry to the CHANGELOG.next.md? Yes

Preview of markdown proposal

[devonakerr]

Looks good to me.

[ebeahan]

There's still the outstanding concern of how to best capture the expected structure of elf.imports.

Defining elf.imports as flattened is straight-forward:

PUT example
{
  "mappings": {
    "properties": {
      "elf": {
        "properties": {
          "imports": {
            "type": "flattened"
          }
        }
      }
    }
  }
}

The outstanding item is what to do about elf.imports.name and elf.imports.type. We cannot explicitly define them underneath imports in the index; Elasticsearch will throw an exception:

"reason" : "unknown parameter [properties] on mapper [imports] of type [flattened]"

We've taken an approach before with objects, like dns.answer, where the docs have the top-level dns.answer field and the expected sub-keys (like dns.answers.class) listed as separate fields. I don't think that approach makes sense for flattened though since there are not actual field mappings associated with each sub-field.

I think our best approach would be describing the expected structure of elf.imports in the field's description. Something like

description: >
  description text....
  
  `elf.imports` expects the following object structure:
  
  {
    "imports": {
      "name": "value",
      "type": "value"
    }
  }

Whatever's decided, we need to add elf.imports to the field definitions. I also made a couple of other notes.

[peasead]

There's still the outstanding concern of how to best capture the expected structure of elf.imports.

Defining elf.imports as flattened is straight-forward:

PUT example
{
  "mappings": {
    "properties": {
      "elf": {
        "properties": {
          "imports": {
            "type": "flattened"
          }
        }
      }
    }
  }
}

The outstanding item is what to do about elf.imports.name and elf.imports.type. We cannot explicitly define them underneath imports in the index; Elasticsearch will throw an exception:

"reason" : "unknown parameter [properties] on mapper [imports] of type [flattened]"

We've taken an approach before with objects, like dns.answer, where the docs have the top-level dns.answer field and the expected sub-keys (like dns.answers.class) listed as separate fields. I don't think that approach makes sense for flattened though since there are not actual field mappings associated with each sub-field.

I think our best approach would be describing the expected structure of elf.imports in the field's description. Something like

description: >
  description text....
  
  `elf.imports` expects the following object structure:
  
  {
    "imports": {
      "name": "value",
      "type": "value"
    }
  }

Whatever's decided, we need to add elf.imports to the field definitions. I also made a couple of other notes.

I just addressed this, please LMK if that meets your concern.

[devonakerr]

Andy, thanks for taking some time to address my questions offline and thanks to both yourself and Eric for your diligence with this RFC. Defining name and type field values should be sufficient to make the elf.imports mappings usable, without going too deeply into the weeds.

[peasead]

Just pinging to get a 2/2 review.

[ebeahan]

Sorry for the delay.

Spotted a few items on a final read-through, but otherwise everything looks good to me.

[devonakerr]

LGTM

[ebeahan]

@devonakerr @peasead @dcode I know we paused on the work around the elf.* fields. However, since the elf.* RFC did make it to stage 2, the fields are now published in the docs. I prefer to avoid indefinitely leaving the elf.* fields at a beta status (and as time goes on, users will adopt them, beta or not 😅 ).

Do we feel comfortable with how these fields are defined now? Are there any reservations for moving to stage 3 and having these fields moving from beta to GA in the schema?

[peasead]

I don't see any outstanding concerns in this Issue; so we can probably move this to Stage 3.

[peasead]

Is there a new sponsor or would you like me to move this?

[ebeahan]

@peasead if you're okay opening the stage 3 PR and if @devonakerr is okay with continuing to sponsor, that'd be great!

[devonakerr]

I heed the call.