Allow configurable time for ES output

[markwalkom]

Here we state;

# Optional index name. The default is "filebeat" and generates
    # [filebeat-]YYYY.MM.DD keys.
    index: "filebeat"

But we don't allow the YYYY.MM.DD to be configurable.

This isn't ideal as users cannot do anything but daily indices without putting Logstash in the pipeline.

[ruflin]

An older discussion about this can be found here: https://github.com/elastic/libbeat/issues/318 It's a good time to close the older one and reference to this one here.

Somehow related: #574

[ruflin]

@markwalkom Is the goal to just make it possible to not have a pattern, or that the output would recognize different output patterns? Can you give some examples?

[markwalkom]

Sure.

Hourly, weekly, monthly indices. Not non-time based though.
On 4 Feb 2016 5:57 pm, "Nicolas Ruflin" [email protected] wrote:

@markwalkom https://github.com/markwalkom Is the goal to just make it
possible to not have a pattern, or that the output would recognize
different output patterns? Can you give some examples?

—
Reply to this email directly or view it on GitHub
#921 (comment).

[ruflin]

That would mean it has the following logic (abstract code)

var index = config.indexname
var pattern = config.pattern

// Pattern not set falls back to default pattern
if pattern == nil {
    pattern = "YYYY.MM.DD"
}


// Empty pattern means pattern is not used
if pattern != "" {

    // Append pattern to index based on timestamp
    pattern = ParsePattern(config.pattern, time.Now())
    index = index + "-" + $PATTERN
}

Would that work, assuming we can interpret common pattern with ParsePattern?

[markwalkom]

Looks OK.

You can do this with LS, not sure how go handles it!
On 4 Feb 2016 6:51 pm, "Nicolas Ruflin" [email protected] wrote:

That would mean it has the following logic (abstract code)

var index = config.indexname
var pattern = config.pattern

// Pattern not set falls back to default pattern
if pattern == nil {
pattern = "YYYY.MM.DD"
}

// Empty pattern means pattern is not used
if pattern != "" {

// Append pattern to index based on timestamp
pattern = ParsePattern(config.pattern, time.Now())
index = index + "-" + $PATTERN

}

Would that work, assuming we can interpret common pattern with
ParsePattern?

—
Reply to this email directly or view it on GitHub
#921 (comment).

[gr0]

Would it be possible to have the index and pattern concatenation part also configurable, so that we could have indices like index_pattern, not only index-pattern?

[markwalkom]

@gr0 you mean _ and - as the separator?

[gr0]

@markwalkom yes, so one could actually have the index pattern that matches the rest of the sources and not ending up with multiple index name patterns or having to modify the rest of the setup to include beats.

For example, if would be nice, to be able to have patterns like [filebeat_]YYYY-MM-DD or [filebeat_]YYYY.MM.DD depending on the needs. Of course with some nice default if the user doesn't want to configure the pattern and just want beats to take care of that.

What do you think?

[UserNotFound]

I'd vote for duplicating how logstash parses this option, at a minimum.

[ruflin]

As a reference here is the way LS does it: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index

@UserNotFound You mention "at a minimum". What are the limitations for you with the LS approach?

[UserNotFound]

@ruflin I have not encountered any limitations, I would just like to be able to name all of my indexes consistently across my environment.

[ruflin]

@UserNotFound Ok. I was asking because if we implement this similar to LS it would be good to know if there are some limitations with the LS approach and we could improve on top of it.

[anhlqn]

Just gave winlogbeat a try today since we are using Solarwinds LEM to gather Windows events. Looking forward for this enhancement to be implemented because:

• I want to limit the number of index/shards for windows event log indexes. Even with 2 shards per indexes, they are still a lot
• Not having to add another layer of LS would save some resources, redundancy issue, and headead ofc

[ruflin]

@markwalkom Could this one help here? elastic/elasticsearch#17814

[markwalkom]

Yes, but only if the user uses the ingest node.

On 27 April 2016 at 21:09, Nicolas Ruflin [email protected] wrote:

@markwalkom https://github.com/markwalkom Could this one help here?
elastic/elasticsearch#17814
elastic/elasticsearch#17814

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#921 (comment)

[ruflin]

@markwalkom So either ingest node or LS can be used, so question is if this feature would still be needed on the beats side (if we not talk about older ES versions).

[markwalkom]

But if you use direct to ES this still matters, so I think it's still a
good option to have.
On 28 Apr 2016 9:09 pm, "Nicolas Ruflin" [email protected] wrote:

@markwalkom https://github.com/markwalkom So either ingest node or LS
can be used, so question is if this feature would still be needed on the
beats side (if we not talk about older ES versions).

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#921 (comment)

[ruflin]

There is now also a new "rollover" API which is also interesting here: https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html

[tsg]

Implemented in #2119.