Add json.* config support in the aws-cloudwatch input

[ndtreviv]

filbeat config:

filebeat.inputs:
  - type: aws-cloudwatch
    log_group_arn: arn:aws:logs:us-east-1:REDACTED:log-group:/ecs/REDACTED:*
    scan_frequency: 1m
    start_position: beginning
    json.keys_under_root: true
    json.add_error_key: true
    json.message_key: message

Cloudwatch logs:

Indexed logs:

...
"message" : """{"timestamp":"2021-06-23T07:27:36.998","level":"INFO","thread":"ForkJoinPool.commonPool-worker-3","logger":"com.mycompany.myapp.RequestHandler","message":"Processing for immediate request status: Progress={jobId=REDACTED, mediaCount=100, mediaProcessed=100, status=PROCESSING, error=null, errorCount=0, started=Wed Jun 16 05:30:52 UTC 2021, finished=null, took=null}","context":"default"}""",
...

I would have expected the json.* properties to kick in, the message field (which is all json) to be parsed as json and the resulting object keys to be put on the document. This includes overwriting the message field itself with the one in the json object.
This works as expected when using a log file input.

For confirmed bugs, please report:

• Version: 7.13.0
• Operating System: N/A (task deployed to ECS)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[ndtreviv]

In case it helps, the logging was written by a springboot application using the following logging config:

<configuration>

    <springProperty scope="context" name="log_level" source="logging.level"
                    defaultValue="info"/>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <!-- encoders are assigned the type
             ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
                <jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter" />
                <timestampFormat>${LOG_DATEFORMAT_PATTERN}</timestampFormat>
                <appendLineSeparator>true</appendLineSeparator>
            </layout>
        </encoder>

    </appender>
    <root level="${log_level}">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

[kaiyan-sheng]

Hi @ndtreviv currently aws-cloudwatch input doesn't support the json.* options. These json options are only for log input in Filebeat. I will leave this issue open to track this enhancement. Thank you!

[ravikesarwani]

Putting it in hold for the time being. Can be looked at once we work on #23575

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!