External ECS fields inside destination index created by transform doesn't map correctly

[kcreddy]

As per the transform's package-spec, the field definition is same as datastream fields. Reference - https://github.com/elastic/package-spec/blob/main/spec/integration/elasticsearch/transform/spec.yml#L55

But, when fields are defined with external (ecs), the mapping in the destination index is incorrect.

Steps to reproduce the bug (found in v 8.9.0 on Elastic Cloud):

1. Define ECS fields under packages/ti_anomali/elasticsearch/transform/latest_ioc/fields/fields.yml as external: ecs just like under packages/ti_anomali/data_stream/threatstream/fields/ecs.yml.

   • Example:

     - name: message
       external: ecs
     - name: error.message
       external: ecs
     

2. Build the package using elastic-package build

3. Install the package by direct upload using zip. https://www.elastic.co/guide/en/fleet/current/fleet-apis.html#installPackageByUpload

4. Check the source datastream mapping and destination index mapping navigating Kibana via:

   • Source: Stack Management -> Index Management -> Component Templates -> logs-ti_anomali.threatstream@package -> Mappings
   • Destination: Stack Management -> Index Management -> Component Templates -> logs-ti_anomali.latest_ioc-template@package -> Mappings

5. Verify the mapping is incorrect.

   • Source:

     "message": {
           "type": "match_only_text"
         },
         "error": {
           "properties": {
             "message": {
               "type": "match_only_text"
             }
           }
         },
     

   • Destination:

     "message": {
           "ignore_above": 1024,
           "type": "keyword"
         },
         "error": {
           "properties": {
             "message": {
               "ignore_above": 1024,
               "type": "keyword"
             }
           }
         },
     

[jsoriano]

I can confirm the issue, indeed elastic-package is not resolving ECS fields in transforms, nor is detecting the external fields on built packages. Moving the issue to elastic-package repo.