Transforms mapping deduction fails for scailed_float types for min and max aggregation

[jeffvestal]

Affected versions:7.2 ->

Problem: Transform deducts the mapping for the destination index based on the configuration, for certain aggregations like min, max transforms takes the mapping of the source index. If the source index uses scaled_float and you configure min and/or max, the transform will fail.

Mitigation: If you run into this issue:

1. create the transform
2. Create the destination index manually with the appropriate mappings
3. start the transform

Original report

Elasticsearch version : 7.5.2
JVM version : openjdk version "1.8.0_222" / and ESS
OS version : Mac and on ESS

Description of the problem including expected versus actual behavior:
When I create a new Transform and I use and aggregation max scaled_float field, I hit 2 issues

1. The mapping for the new index is not correctly created throwing an error Field [max] misses required parameter [scaling_factor] and it won’t create the index
2. After manually creating the mapping, it starts but eventually fails with Caused by: com.fasterxml.jackson.core.JsonParseException: Current token (START_OBJECT) not numeric, can not use numeric value accessors

This happens on both ESS and local Mac install

Steps to reproduce:

Create source index mapping

PUT test_source
{
    "mappings" : {
      "properties" : {
        "trace" : {
          "properties" : {
            "id" : {
              "type" : "keyword"
            }
          }
        },
        "labels" : {
          "dynamic" : "true",
          "properties" : {
            "step" : {
              "type" : "scaled_float",
              "scaling_factor" : 1000000.0
            }
          }
        }
      }
    }
}

Put a few source docs abc,1 abc,2 abc,3 def,1 def2

POST test_source/_doc
{
  "trace.id": "abc",
  "labels.step": 3
}

##Create transform

PUT _transform/test_dest_01
{
  "source": {
    "index": [
      "test_source"
    ],
    "query": {
      "match_all": {}
    }
  },
  "pivot": {
    "group_by": {
      "trace.id": {
        "terms": {
          "field": "trace.id"
        }
      }
    },
    "aggregations": {
            "labels.step.max": {
        "max": {
          "field": "labels.step"
        }
      }
    }
  },
  "dest": {
    "index": "test_dest_01"
  }
}

attempt to start but fail

POST _transform/test_dest_01/_start
{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Failed to parse mapping [_doc]: Field [max] misses required parameter [scaling_factor]"
      }
    ],
    "type": "runtime_exception",
    "reason": "runtime_exception: Could not create destination index [test_dest_01] for transform [test_dest_01]",
    "caused_by": {
      "type": "mapper_parsing_exception",
      "reason": "Failed to parse mapping [_doc]: Field [max] misses required parameter [scaling_factor]",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "Field [max] misses required parameter [scaling_factor]"
      }
    }
  },
  "status": 500
}

Put dest mapping manually

PUT test_dest_01
{
    "mappings" : {
      "_meta" : {
        "created_by" : "transform",
        "_transform" : {
          "transform" : "thtest_dest_012",
          "version" : {
            "created" : "7.5.2"
          }
        }
      },
      "properties" : {
        "trace" : {
          "properties" : {
            "id" : {
              "type" : "keyword"
            }
          }
        },
        "labels" : {
          "dynamic" : "true",
          "properties" : {
            "step" : {
              "type" : "scaled_float",
              "scaling_factor" : 1000000.0
            }
          }
        }
      }
    }
}

Response

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "test_dest_01"
}

Start again

POST _transform/test_dest_01/_start`
`
{
  "acknowledged" : true
}

Status

GET _transform/test_dest_01/_stats
-> starts but never completes

Provide logs (if relevant):

Eventually fails after 10 bulk exceptions

[2020-01-31T16:03:38,130][DEBUG][o.e.a.b.TransportShardBulkAction] [critter.lan] [test_dest_01][0] failed to execute bulk item (index) index {[test_dest_01][_doc][ZLnd4QnTNZcP1r6XC5Q9jaEAAAAAAAAA], source[{"trace":{"id":"def"},"labels":{"step":{"max":2.0}}}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [labels.step] of type [scaled_float] in document with id 'ZLnd4QnTNZcP1r6XC5Q9jaEAAAAAAAAA'. Preview of field's value: '{max=2.0}'
    at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:299) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:418) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:418) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:791) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:768) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:740) ~[elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:258) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:161) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:193) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:118) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:917) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:108) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:394) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:316) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$22(IndexShard.java:2796) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2770) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:858) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:312) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:275) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.5.2.jar:7.5.2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) [x-pack-security-7.5.2.jar:7.5.2]
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:752) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: com.fasterxml.jackson.core.JsonParseException: Current token (START_OBJECT) not numeric, can not use numeric value accessors
 at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@6b657c5c; line: 1, column: 41]
    at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11]
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.11.jar:2.8.11]
    at com.fasterxml.jackson.core.base.ParserBase._parseNumericValue(ParserBase.java:837) ~[jackson-core-2.8.11.jar:2.8.11]
    at com.fasterxml.jackson.core.base.ParserBase.getDoubleValue(ParserBase.java:751) ~[jackson-core-2.8.11.jar:2.8.11]
    at org.elasticsearch.common.xcontent.json.JsonXContentParser.doDoubleValue(JsonXContentParser.java:176) ~[elasticsearch-x-content-7.5.2.jar:7.5.2]
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.doubleValue(AbstractXContentParser.java:243) ~[elasticsearch-x-content-7.5.2.jar:7.5.2]
    at org.elasticsearch.index.mapper.ScaledFloatFieldMapper.parse(ScaledFloatFieldMapper.java:474) ~[?:?]
    at org.elasticsearch.index.mapper.ScaledFloatFieldMapper.parseCreateField(ScaledFloatFieldMapper.java:396) ~[?:?]
    at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:277) ~[elasticsearch-7.5.2.jar:7.5.2]

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[hendrikmuhs]

@jeffvestal

Thanks for the report and the detailed step to reproduce, very helpful!

I found a problem in your mappings. You configured the transform with an output field labels.step.max however, your mapping configuration maps labels.step to scaled_float instead of labels.step.max. With other words your mapping configuration lacks another level of nesting.

If you configure your mapping like this, your example should work:

PUT test_dest_05
{
  "mappings": {
    "properties": {
      "trace": {
        "properties": {
          "id": {
            "type": "keyword"
          }
        }
      },
      "labels": {
        "properties": {
          "step": {
            "properties": {
              "max": {
                "type": "scaled_float",
                "scaling_factor": 1000000
              }
            }
          }
        }
      }
    }
  }
}

So the answer to your found isues:

Issue 1: mapping deduction for scaled float fails due to omitting scaling_factor

Issue 2: error handling of mapping errors, in #51307 we addressed this already. Starting in 7.7 the transform won't retry 10 times and the error message will contain a hint about the mapping conflict

Therefore issue 1 remains, I will change the title accordingly.

[jeffvestal]

@hendrikmuhs Thanks for the info on the mapping, good catch on that.

I should have included the original use case.
Using the apm-* index and original dynamic mapping template (created by apm) we are writing a value to labels.step, which gets mapped to a scaled_float automatically.
"labels" : { "dynamic" : "true", "properties" : { "step" : { "type" : "scaled_float", "scaling_factor" : 1000000.0 } } }

I'm using transforms to, among other things, get the max(labels.step). Originally when I started the transforms job, it wasn't able to create the destination index so I created the mapping manually to test if the job it would still fail (though as you pointed out I missed a nesting!)

You messaged me your solution to falling back to using a float in the transform, which would be fine in my case at least.