Default value for sprintf interpolation (Feature)

[breml]

In a lot places, Logstash allows to reference fields with the sprintf format(https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#sprintf). Unfortunately if the referenced field does not exist, sprintf defaults to leave the the reference untouched, which may lead to very unsatisfactory results. Therefore I suggest to allow a default value for the sprintf interpolation, e.g.

output {
  file {
    path => "/var/log/%{type:-default}.%{+yyyy.MM.dd.HH}"
  }
}

As delimiter between the field reference and the default value I suggest :- (taken from bash, unlikely to exist in a Logstash event field name).

[ph]

I really like the idea of having a way to define a default value for a key in the string interpolation, this has come to my mind when I did the refactor and optimization of that part of the code.

I am not sure if the :- syntax is fine in that case, the other option that I could think of is this django-like template syntax %{type|default:mydefault}. Its a bit more explicit? @jordansissel @jsvd any opinion on this?

Another common request is a way to access environment variables through string interpolation, lets take both cases into consideration to come up with a the best syntax.

If i have an exported variable ABC="hello world" and I would like to access it via string interpolation maybe we could do the following

%{ENV|ABC|default:logs}%

What do you think? This would allow us to be flexible with new features and | is rarely used for keys?

[breml]

I don't have any preference for the syntax. Actually I used the pipe (|) my self in the first drafts.

Regarding the environment variables have found #3944 which it self references logstash-plugins/logstash-filter-environment/pull/5. I think all the cases regarding environment variables, which could be addressed with the combined string interpolation syntax proposed by @ph, could also be solved by logstash-filter-environment. Therefore I suggest to not overload this pull request.

[ph]

@breml I am +1 to not overload the PR, just wanted to make sure we could grow from it :)

[jordansissel]

As a workaround, you can usually do this:

filter {
  ...
  if ![type] {
    mutate { add_field => { "type" => "default" } }
  }
}

output {
  file {
    path => "/var/log/%{type}.%{+yyyy.MM.dd.HH}"
  }
}

[jordansissel]

I agree it would be nice to have more functionality in %{...} syntax, but I'm not sure how to approach it. There's more than just default values we may want to add (possible post-processing of values? I don't know). I'm hesitant to add more functionality into %{} for just one feature when there are lots of features we could add, and I'm worried about how this can grow in a way that is sensible for users.

[breml]

@jordansissel: Thanks for the workaround, we will try this one. Up until now we used the blacklist_values feature of https://github.com/logstash-plugins/logstash-filter-prune to remove fields with %{...} content. But the above workaround is sure the better and cleaner solution.

[breml]

I agree with @jordansissel, to have a clean syntax for a possible extension of the %{...} syntax, some more thinking is needed. I just proposed a solution for my actual problem, without the overall architecture in mind.

I am not sure, if we have to invent the wheel again. There are similar string interpolation features implemented in other places/languages.
bash, where I took the syntax from, is one of them. Maybe we should have a look at this solutions to get a better feeling, where a possible extension of the %{...} could lead us.

[breml]

My PR #4417 is closed, based on the discussion in this issue. My question is: How do we precede with this one?

[andrewvc]

@suyograo what was wrong with the code in #4417 ? I think it actual makes sense (though I'd like the delimiter to be |.

[suyograo]

It looks like we haven't agreed on the syntax to define defaults based on above comments.

bash, where I took the syntax from, is one of them. Maybe we should have a look at this solutions to get a better feeling, where a possible extension of the %{...} could lead us.

I personally liked what @ph proposed. Its more explicit - %{type|default:mydefault}

[ph]

I just reread the issue:

This comment from @jordansissel

possible post-processing of values

Like type coercion? We could implement it that way:

%{type|default:1,convert:integer}

not sure if it would be better if we support quoted text too, this would make the syntax more resilient to future changes.

%{type|default:"my new default"}

[suyograo]

@breml How about sticking to the syntax %{field:"my new default"}. This is inline with the syntax we introduced for expanding environment variables: #3944. Thoughts?