Show log entries based on time range

[daschr]

Hi!

Is your feature request related to a problem? Please describe.
I have very large log files (>1,000,000 lines, ~50,000 new lines per day) and sometimes need to view older log entries from days ago.

However, seeking through these logs with the default JSON-FIle based logging driver using docker logs --since / --until takes a long time. Thus, I've written an own logging driver based on sqlite3 databases (https://github.com/daschr/logsqlite) which allows much faster time range-based lookups.

But reading the logs in the terminal is also not so nice as having a webinterface like dozzle.
Unfortunately, dozzle only allows to scroll up until it load older entries.

Describe the solution you'd like
It would be great if one can specify a time range (--since and --until) or starting time (--since) using a time and date picker.
Then, dozzle shows the entries of that time range.

[amir20]

Hi @daschr. First of all, that's an impressive driver implementation. I wish I could do more Rust but haven't had time.

Dozzle's principle has been simplicty and focus on debugging by streaming in real-time. Due to that reason, I haven't invested a lot of time in searching or going back to a random time in history. Dozzle is more about streaming than "random access" use cases.

There had been a lot of discussion adding similar functionalities to Dozzle by implementing a sqllite solution too. Ultimately, I felt it was beyond Dozzle's focus. I felt that would be a lot of up keeping not worth doing.

However, you raise an interesting point because now Dozzle can just be the viewer and have the log driver to the dirty work of making random access faster. My only issue is that it would be a lot of work to go to a specific location in the logs. Perhaps, there is a compromise.

What is exactly your use case? Could something like scroll back 24 hours be useful? I was thinking to stay aligned with the "streaming" principle, it could be possible when the user scroll up, they could click a button to scroll beyond a day. This is still complicated because most containers probably don't have that much data.

What do you think?

[daschr]

Hi @amir20! Thank You for the kind words.

Ah I see, Dozzle is not designed for pagination but for showing the latest entries.

Yes, mostly I just scroll back to the previous day, which currently takes a long time in Dozzle because it it loads only a few minutes ago at each step.
It would be awesome if one could jump a day back!

Again, thank You for this project. It is the best tool I have tested so far.
Cheers!

[amir20]

@daschr So I thought about this a little more. It seems more complicated than I thought.

If I was to offer scroll back 24 hours then one would expect that scrolling down would continue the stream from that day. But right now it goes back to real time which is the tip of the log. Implementing this scroll buffer both directions would be hard.

So then I thought, ok let's just offer search between two dates, like you suggested. But as you said, some may have millions of lines. And the UI would have to be smart enough to start paginating between these selected dates. Which goes back to the first problem again.

Therefore, I think this is harder than I thought to implement. I have to think about a better way to allow some buffer up and down for it to work well.

Probably won't be something I can work in the short term.

[amir20]

I have thought about this feature more. And a lot would need to change for it to be able to jump up to a random location. Everything in Dozzle is designed to be continuous stream of logs. Loading previous content just loads what is needed to scroll up.

I am going to close this as won't do for now. But will come back to it or someone else can help.

[amir20]

I am going to reopen this. I think I have a better idea. Instead of dealing with live stream and going back in time, it makes more sense for Dozzle to stop streaming live logs when viewing previous logs. This makes it a whole a lot simpler. I haven't really thought about the code yet but it's worth keep it open.

[daschr]

I agree that this would be much better. But i think you may not let the user specify a interval but just a timestamp.
Then, you can display all lines beginning from that timestamp to n minutes after that. Where n is a number the user can select in the webinterface (f.e using a drop-down menu with 1, 2, 5, 15, 30 minutes).

For the pagination, maybe this inspires you:

You may display a button named "previous n minutes" or something when a user scolls up.
If this button is pressed, the previous n minutes older lines are displayed with a button named "next n minutes" below. If "next" is pressed, the n minutes younger lines with a timestamp newer than the youngest one in the current page are displayed.

[EDIflyer]

Really interesting idea - agree would be nice to pick a time period to view logs for.

Related to this - I was also playing about looking at the functionality to look at log level (#2432) - I realised that when searching some logs nothing is return but that will just be because you're (quite reasonably) only loading the most recent x number of log lines.

I wanted to go back to look at an older error in npm and found Portainer has quite a nice option that lets you specify the number of lines loaded - I wasn't sure if that might be something useful for Dozzle too? I didn't want to confuse things by opening another Github issue as I feel it's more related to this and the log level one but thought it worth mentioning as a potential idea. I still wouldn't expect it all to be loaded for instant scrolling, it was more for the search to work but I'm not sure from an internal perspective if that's possible or if Dozzle only searches within the lines in the display buffer.

As others have said Dozzle is simply brilliant, I think it also has one of the cleanest issue and PR pages on Github, you keep impressively on top of everything, @amir20!!

Default 100 lines

Increased to 10,000 lines to find an old warning

[amir20]

Thanks @EDIflyer. I hadn't seen your comment before. I ended up installing Portainer and played around with it a little. The logging page is almost unusable because with 10k logs it goes up to 1GB of mem. I am not sure if Portainer is doing something to clean up.

I am still thinking about this feature. I am thinking of just having some kind of slider where you can go to a specific date/time. I hope to come back to this.

[FrancYescO]

just to bump this issue... portainer got the logs page a little renewed, not big changes but a little better