Support other duration formats than Go duration format

[cykl]

Is your feature request related to a problem? Please describe.

LogQL only support Go duration format. Applications not written in Go are unlikely to produce durations in Go duration format which lead to a degraded query experience.

• If logs use ISO 8601 duration, there is no way to filter on a duration.
• If logs use a standard time unit (ex. Elastic Common Schema use nanosecond), users have to write something like duration > 35000000000 rather than duration > 35s which is cumbersome and error prone.

Describe the solution you'd like

I had like to be able to parse common duration representations (ISO 8601 duration, fractional (seconds|milliseconds) etc.).

Describe alternatives you've considered

• Update applications to use Go duration format

Additional context

I haven't had time to evaluate all the consequences of this request nor to devise a solution. However, I'm sure this issue will itch enough users to worth an anemic issue.

[cyriltovena]

This is an acceptable feature request but lack of example on how we can achieve this.

The left side should be explicit, some examples could be:

• | some_iso_date > iso_date("2007-03-01T13:00:00Z")
• | some_iso_duration >= iso_duration("P3Y")

What do you think ? any better name ?

should we try to infer the type without the function ? like this:

• | some_iso_date > 2007-03-01T13:00:00Z
• | some_iso_duration >= P3Y

Not sure yet of all of this I think we need to design this correctly first, specially that we want to probably support more format in the future, I expect people asking for other date and duration format.

[cykl]

This is an acceptable feature request but lack of example on how we can achieve this.

Definitely. I was hopping you already had some plan 😉

What do you think ?

Note: I have not read relevant Loki source code yet, so my words may make no sense at all.

My understanding is that ... | myField > 1m15s | ... states: "parse myField as a duration and compare it to this duration constant".

Using Go duration format to express the constant part seems perfectly fine. An user might prefer one duration format or some other but it's mostly nitpicking. It takes 10s to understand how Go duration format works.

The main issue is how to convert myField into a duration that can be compared to the constant. All your proposals change how to express the constant which surprised me. I would have thought that you would propose to use label_format and introduce dedicated template functions. Something like:

... 
  | label_format myField_go={{ .myField | parseIsoDuration }} 
  | myField_normalized > 1m15s 
  | ...

I have not played much with label_format yet but I think it would do the trick. However, having to use a label_format for a such common operation might be too cumbersome. Maybe we can find some shorter syntax like:

... 
  | {{ .myField | parseIsoDuration }} > 1m15s
  | ...

If I don't plan to reuse the label for another purpose it seems both concise and allows to leverage template functions.

What do you thing?

[cyriltovena]

I think it's a good idea to start with for sure template function are easier to add.

However template are hard to scale it's the most inefficient stage in the language currently but we can solve that at some point.

[Mario-Hofstaetter]

@cykl the issue title is missing the word "other" I think.

@cyriltovena Would it be possible to enable sorting / filtering a string alphabetically for this?
We are using the default ToString representation of the .NET Timespan which gets you (dummy-) log messages like these:

2021-02-08 10:18:53 Finished 'more util' in 00:00:00.1032464
2021-02-08 10:18:48 Finished 'more util' in 00:00:02.2612968

While not as user friendly as Duration > "1s" I tried using the Greater Than operator but unfortunately it fails for string:

{Service="storageservice"} |= "Finish" | json | Duration > "00:00:01"

Of course, Culture behavior (cAsE, accent sensitivity ...) is a big can of worms here.

Would you deem this worthy its own issue? I can think of some use cases for alphabetically sorting strings (e.g. postal codes that are not numeric)

[cyriltovena]

Yeah but let's create another issue for this then.

[kavirajk]

This is interesting! +1

Just putting the possible options based on above discussions.

So the use case is basically to compare time and duration in different format. say,

... | label op "(date or duration value") | ...

Q: Currently LogQL support duration type not date. Do we have to support date format as well? (that can also taken as completely new feature and tracked on separate issue)

IMO, keeping single type of format on right hand side of the op is much simpler and less confusing for users of LogQL as well.

Assuming, we keep right hand side like as it is now (Duration format with Go style.), all we need to do is somehow make op recognise the label duration format (say if label value is in ISO duration format)

And I think recognising this label in ISO format implicitly is better. Why?

1. No extra syntax
2. NO extra learning
3. Easy to add support for more format without introducing any visible changes (just have to record it in the doc)

Examples

| some_iso_duration >= 2h3m

| some_go_style_duration >= 2h3m

| some_style_invented_yesterday_duration >= 2h3m

I also think template seems like overkill for this.

[kavirajk]

After discussion with @cyriltovena, looks like there is some real problem in making LogQL grammar to understand the label duration type implicitly.

Here is the another possible approach.

Basic idea is to extend current duration() function that we currently use only in unwrap expressions on metric queries.
With this approach now duration() will be available in non-unwrap expressions as well.

We can add optional argument to duration(<duration_string>, <from_format>, <to_format)

from_format - represents the duration format of the label
to_format - represents the duration format to convert to

Both from_format and to_format are optional. In that case, its behaviour is same as now (takes Go style duration)

if from_format is provided, it assumes the given string is in from_format and one can use it for comparition operators.
If to_format is provided, it will try to convert the duration into to_format. One cannot use to_format without from_format

Scope:

1. duration(duration_string, format)match labels with different duration format - via label filters
2. duration(duration_string, from, to) format label values to different duration format - via label formatter

Examples:

1. Normal case, no conversion function, label is already in go duration format

...| response_duration < 2m30s | 

2. Label is in iso format

...| response_duration < duration("P3Y", "iso") | 

3. Create new label with go duration from iso duration format via label formatter

...| label_format go_duration=duration(response_duration, "iso", "go") | 

Here we considered only "go" and "iso" format style, but we can extend it into different format types like ".net", "java", "ruby" etc.. on demand.

In future we can do same for date. But that will be completely new feature and out-of-scope here.