Status query syntax

[Nukesor]

I started implementing the pueue status query on this branch: https://github.com/Nukesor/pueue/tree/filter-status
It's still very much work-in-progress, but it's starting to shape into something usable.
The main reason why I'm writing this ticket is that I want some feedback on the current syntax.

A small upfront disclaimer:
The first draft of the syntax will be experimantal. If it get's released, there'll be a big note on top of it that the syntax might become subject to change and that it shouldn't be considered stable!

Furthermore, I don't plan to make the syntax feature-complete from the get-go. The idea is to implement the basic features and continue working on it with real-world feedback and usecases.

Okay, now that that's out of the way here's the current syntax.

There're currently three sections:

Column selection:

This is a basic filter on which columns should be visible

• columns=id,status,start,end

Filters

This is where it get's a bit more complex.
For now a few filters are planned:

• label [=|!=] string: Filter for a specific label
• [start|end|enqueue_at] [=|!=|<|>] [YYYY-MM-DD | YYYY-MM-DD hh:mm | hh:mm] Filter any of the three date related fields.
• status [=|!=] [queued|stashed|running|paused|done|failed] Filter tasks based on their status

Ordering

• order_by [any_column] = [asc|desc] A simple ordering depending on a column and a direction.

Limit

• [first|last] [count] Only show the count first or last tasks.

Complete query

A complete query has this structure:
[column_selection]? [filter]*? [order]? [limit]?

Examples:

• columns=id,status,command,end label=special start>2022-09-01 order_by status desc
• label=special
• start>10:30 end<18:00 status=success first 10

Questions:

• What do you think of this syntax?
• Is there any fundamental missing feature?
• Would this cover your 80% usecase?

Further Additions to the syntax

• A "contains" filter for labels. The operator could be a %=

[Nukesor]

Ping @codekoala @d33tah @kjhoerr @stranger-danger-zamu

[Nukesor]

I just finished the implementation of all features I deemed "essential".

It would be great if you guys could give it a test run based on your common scenarios :)

cd /tmp
git clone [email protected]:Nukesor/pueue
cd pueue
git checkout filter-status
cargo install --path . --locked

[codekoala]

@Nukesor this looks fantastic! Thank you for implementing this feature. I will give it a spin this coming week and report back with any feedback.

[mjpieters]

I like this idea! I'd try to keep this simple; the --json output gives scripters enough power to implement more complex behaviour.

I do wonder if there is a good use-case for substring matching, at least for labels. E.g. for my current needs I have [name]/[identifier]-[isodate] strings that echo the destination of the download, and I use that information in my jq scripts to group by year or by name. A simple 'contains' test would do there, or if you want to go really fancy, support regex matching?

Another queue is labelled with the number of items to process (as the log only shows me what numbered files have been processed, so I can compare label and current file number to see how far along the job is), so a stretch goal would be for <, <=, >, and >= to work with labels too. If need be, just make that use lexicographical ordering, and I'll just zero-pad the labels (images: 123 is more than images: 042). This does feel a little bit like it would push the syntax into becoming complex however.

[Nukesor]

I agree that the syntax should be kept rather simple :) This is also the reason why I didn't introduce the OR keyword (or something alike) yet, as this make things so much more complex. Let's decide whether something is feasible per use case.

Adding >|=>|<|<= for lexical ordering in labels shouldn't be too much of a problem as the complex comparison stuff is already shipped with Rust. As a matter of fact, you can already order tasks lexically by labels.

I'm also not against a contains operator. Do you know of any intuitive symbols for this kind of operation? I found ~, but I also associate this with the NOT operator :D.

Regarding regex support, I'm not generally against it, but I would like to keep it simple for the first iteration of the syntax.

[mjpieters]

I'm also not against a contains operator. Do you know of any intuitive symbols for this kind of operation? I found ~, but I also associate this with the NOT operator :D.

How shell friendly does it need to remain; more friendly than find operators, presumably ;-) ~ is, in my mind, also more of a REGEX operator (notably, AWK, but also PostgreSQL). And in C and the many languages derived from it, ~ is a bitwise NOT operator.

Looking at other contexts for similar use-cases I peeked at CSS attribute selectors; ~= there is reserved for matching a whole word in a white-space separated sequence of words, and *= is for substring matches. I was hoping for the latter here. Of course, * is a shell meta-character, so that's a downside.

I do like the idea of combining a symbol with = here[symbol]= or =[symbol]. PERL introduced =~ for regexes, and that has been adopted widely (Ruby, Groovy, CSS uses the inverted combo, etc.). And since you are emulating SQL here with the expression ordering, perhaps %, %= or =% would be a good spelling for a contains operator? I'd lean towards %= for intuitive reasons, I'll have to come up with better reasoning for why I like that better than =% or %..

E.g.:

label%=2022-01-

would filter the output to only show items with January 2022 in the label.

[Nukesor]

How shell friendly does it need to remain; more friendly than find operators, presumably ;-)

The current operators > and ! already aren't shell friendly at all :D.
Queries should pretty much always be enclosed in a quotes.

I'd lean towards %= for intuitive reasons, I'll have to come up with better reasoning for why I like that better than =% or %..

I like %= quite a bit. A standalone % is rather confusing as it reminds of the modulo operator. %= however should be rather intuitive for people that know SQL and the = makes it clear that there's at least some kind of similarity check going on.

*= is for substring matches. I was hoping for the latter here. Of course, * is a shell meta-character, so that's a downside.

* being a meta-character isn't much of a problem, as there already are a few of those in the current draft of the syntax.
But, to be honest, I don't really have any association with *=... Guess I don't do enough CSS stuff

[Nukesor]

I'll try to keep the base syntax as it is for now, further operators and filters will be added in follow-up PRs to keep the scope manageable.
The current PR already has 1500+ added lines regarding filters.

Everything should be rather well tested now, but I'll keep the PR open for another 1-2 weeks for people to test :)