Revise deletion of terms and goups from multi-level complex expressions

[Betterbird]

See #62 (comment).

[ThiloteE]

Deleting a term or group out of the middle of a complex expression do not (yet) adjust the counts of beginning/ending groups, so deletion in complex situation will likely cause invalid rules.

[Betterbird]

Concrete examples:

Delete term "c" and re-insert. That causes damage.

Deleting any of "a", "b", "c" or "d" causes damage.

Of course these are all far-fetched examples with multiple begins/ends on the same term. Operations on the simple rules like (a && b) || (c && d) or (a || b) && (c || d) work fine.

[Betterbird]

Implemented a check so the invalid rule doesn't go undetected.

[Betterbird]

@RealRaven2000: Note that in Issue #62 we implemented balancing begin and end of groups, that is the number of [ always matches the number of ] in the msgFilterRules.dat files.

Rules that were created previously where one term terminated multiple groups will now be flagged as error.

[RealRaven2000]

@RealRaven2000: Note that in Issue #62 we implemented balancing begin and end of groups, that is the number of [ always matches the number of ] in the msgFilterRules.dat files.

Rules that were created previously where one term terminated multiple groups will now be flagged as error.

Good to know - that would be an edge case as they would only be created with the previous beta versions anyway.

[Betterbird]

This is not correct. Since we first shipped the feature in BB 91.9.0 at the end of April 2022, multi-level groups can be created.
Formerly a && (b && (c && d) was stored as

condition="AND (subject,contains,a) AND [subject,contains,b) AND [subject,contains,c) AND (subject,contains,d]"

now it is stored as

condition="AND (subject,contains,a) AND [subject,contains,b) AND [subject,contains,c) AND (subject,contains,d]]"

Editing the former will now produce an error although rule execution still works.

[ThiloteE]

Maybe the law of distributivity could help making things easier:

∨ (logical or)
∧ (logical and)

In particular, I was thinking that even very complex filters could be replaced by ones that arguably are still very complex, but might have less problems with left over braces after deletion. A change of state would need to be replicated across multiple sub-formulas, but it can be done and is definitely logically sound. An example for this can be seen in the following screenshot:

Here are some more examples:

1. Simple examples:

   Deletion of a; starting from distributive syntax
   a ∨ (b ∧ c) <=> (a ∨ b) ∧ (a ∨ c) --> ( v b) ∧ ( v c) --> (b) ∧ (c) --> b ∧ c
   a ∧ (b ∨ c) <=> (a ∧ b) ∨ (a ∧ c) --> ( ∧ b) v ( ∧ c) --> (b) v (c) --> b v c

   As one can see above, we got rid of all braces! I have refrained from removing braces from sub-formulas that contain multiple elements. I assume, there is no risk to get rid of braces that only cover a single element.

   One can see in the following example that we will have some left over braces if we do the same for a ∨ (b ∧ c) and a ∧ (b ∨ c):

   deletion of a; starting from normal syntax
   a ∨ (b ∧ c) --> v (b ∧ c) --> (b ∧ c)
   a ∧ (b ∨ c) --> v (b v c) --> (b v c)

   Here it is even necessary to remove v from outside braces, therefore

   Not sure if this proposed syntax using the law of distributivity is enough, though. The more complex examples down below show, there are still situations where superflous braces will exist.

2. More Complex example:

   Deletion of P
   ((P ∧ Q) v (R ∧ S)) <=> ((P v R) ∧ (P v S)) ∧ ((Q v R) ∧ (Q v S)) --> (( v R) ∧ ( v S)) ∧ ((Q v R) ∧ (Q v S)) --> ((R) ∧ (S)) ∧ ((Q v R) ∧ (Q v S)) --> (R ∧ S) ∧ ((Q v R) ∧ (Q v S)) --> (R ∧ S) ∧ ((Q v R) ∧ (Q v S)). Ideally: R ∧ S ∧ ((Q v R) ∧ (Q v S))

   • some braces are left, but I guess they do no harm?

3. Another complex one:

   Deletion of c
   a v [b ∧ (c v d v e)] v f <=> a v [(b ∧ c) v (b ∧ d) v (b ∧ e)] v f --> a v [(b ∧ ) v (b ∧ d) v (b ∧ e)] v f --> a v [b v (b ∧ d) v (b ∧ e)] v f

   • no problems here! Works out well!

4. Even MORE complexity:

   Deletion of c:
   a ∧ (b ∧ (c ∧ (d ∧ e))) ∧ f <=> and at this point I realized to do the transforming from the left side to the right side would be UTTERLY COMPLEX! haha ok, maybe this distributive syntax is not such a good idea after all? Unless computers can do this thing.

   • was not able to do it. My brain is melting.

Nevertheless, regardless of syntax, to detect a change of state, making use of identity elements could be worth a shot?

• "Let (S, ∗) be a set S equipped with a binary operation ∗. Then an element e of S is called a left identity if e ∗ s = s for all s in S, and a right identity if s ∗ e = s for all s in S. If e is both a left identity and a right identity, then it is called a two-sided identity, or simply an identity." - Source: Wikipedia (2022-07-31)

• Left or right Identity therefore could be the implied basis for an automatic filter change.
  E.g.
  

Another method to detect a change of state could maybe be based on truth tables:

Additional related literature:

https://en.wikipedia.org/w/index.php?title=Boolean_algebra&oldid=1098861709
https://en.wikipedia.org/w/index.php?title=Boolean_algebra_(structure)&oldid=1060326535
https://en.wikipedia.org/w/index.php?title=Identity_element&oldid=1100646601
https://en.wikipedia.org/w/index.php?title=Boolean_differential_calculus&oldid=1087661815

[ThiloteE]

Reminder for myself about Filter Syntax:

Syntax for Thunderbird 91.12 / 102 (apparently?): a && (b && (c && d)
Syntax for Betterbird 91.9 / 91.10: condition="AND (subject,contains,a) AND [subject,contains,b) AND [subject,contains,c) AND (subject,contains,d]"
Syntax for Betterbird 91.11 / 91.12 /102 ... : condition="AND (subject,contains,a) AND [subject,contains,b) AND [subject,contains,c) AND (subject,contains,d]]"

Brainstorming about parsing and transformations:

Simply better parsing of filter-rules and automatically converting erroneous rules to better ones in the background of course could work too.

So I was thinking a little bit and maybe the following helps for brainstorming:

The following one is probably the most interesting one, since this is the one that refers to the first screenshot in #63 (comment)

trigger deletion of c.
a && [b && [c && [d && e]]] --> a && [b && [ && [d && e]]] --> a && [b && && [d && e]] --> a && [b && [d && e]]
a && [b OR [c && [d OR e]]] --> a && [b OR [ && [d OR e]]] --> a && [b OR && [d OR e]] --> a && [b OR [d OR e]]
a && [b OR [c OR [d && e]]] --> a && [b OR [ OR [d && e]]] --> a && [b OR OR [d && e]] --> a && [b OR [d && e]]
a && [b && [c OR [d OR e]]] --> a && [b && [ OR [d OR e]]] --> a && [b && OR [d OR e]] --> a && [b && [d OR e]]
Required mechanisms: detect "&& [ &&", "OR [ &&", "OR [ OR", "&& [ OR"; detect BOTH superflous [ and ]; Detect "&& &&", "OR &&", "OR OR", "&& OR"

Additional:

trigger deletion of c:
a && (b && (c && d) --> a && (b && ( && d) --> a && (b && && d) --> a && (b && d)
a && (b OR (c && d) --> a && (b OR ( && d) --> a && (b OR && d) --> a && (b OR d)
a && (b OR (c OR d) --> a && (b OR ( OR d) --> a && (b OR OR d) --> a && (b OR d)
a && (b && (c OR d) --> a && (b && ( OR d) --> a && (b && OR d) --> a && (b && d)
Required mechanisms: detect d is "alone"; detect &&, "OR" are superfluous; detect "&& &&", "OR &&", "OR OR", "&& OR"

trigger deletion of d:
a && [b && [c && d && e]] --> a && [b && [c && && e]] --> a && [b && [c && e]]
a && [b OR [c && d && e]] --> a && [b OR [c && && e]] --> a && [b OR [c && e]]
a && [b OR [c OR d OR e]] --> a && [b OR [c OR OR e]] --> a && [b OR [c OR e]]
a && [b && [c OR d OR e]] --> a && [b && [c OR OR e]] --> a && [b && [c OR e]]
Required mechanisms: detect "&& &&", "OR &&", "OR OR", "&& OR"

trigger deletion of c.
a && [b && [c && d]] --> a && [b && [ && d]] --> a && [b && && d]] --> a && [b && && d] --> a && [b && d]
a && [b OR [c && d]] --> a && [b OR [ && d]] --> a && [b OR && d]] --> a && [b OR && d] --> a && [b OR d]
a && [b OR [c OR d]] --> a && [b OR [ OR d]] --> a && [b OR OR d]] --> a && [b OR OR d] --> a && [b OR d]
a && [b && [c OR d]] --> a && [b && [ OR d]] --> a && [b && OR d]] --> a && [b && OR d] --> a && [b && d]
Required mechanisms: detect d is "alone"; detect ] is superfluous; detect "&& &&", "OR &&", "OR OR", "&& OR"

trigger deletion of [c. (actually trigger deletion of c, but detect that it is the first value in the chain, therefore trigger deletion of [c)
a && [b && [c && d]] --> a && [b && && d]] --> a && [b && && d] --> a && [b && d]
a && [b OR [c && d]] --> a && [b OR && d]] --> a && [b OR && d] --> a && [b OR d]
a && [b OR [c OR d]] --> a && [b OR OR d]] --> a && [b OR OR d] --> a && [b OR d]
a && [b && [c OR d]] --> a && [b && OR d]] --> a && [b && OR d] --> a && [b && d]
Required mechanisms: detect c is next to [; detect ] is superfluous; detect "&& &&", "OR &&", "OR OR", "&& OR"

Of course, there are a lot more examples one could come up with...

[Betterbird]

Thanks for your enthusiasm. Three things:

1. Right now, we're in holiday mode.
2. We might pick this up again when version 91 is deprecated, so we won't have to maintain two versions with changes.
3. The internal storage of search terms is unfortunately a list, not an expression tree. So all the transformations you suggest would need to be done on a tree 😢 😭. What's missing for deletion is to balance the opening/closing brackets.

[ThiloteE]

Don't worry, have a blast :-) There needs to be some you time too :-)

[CapitainFlam]

2. We might pick this up again when version 91 is deprecated, so we won't have to maintain two versions with changes.

Hi @Betterbird !

Newcomer here. I came 'coz i'd like to install Thunderbird (i'm using FireFox since ...forever, as I was under Netscape, but never switched my mail client, still gmail)... And I think i'll switch to Betterbird !

So whatever, long story short, after having read pages and pages of your website and different bug reports (including this one), I think that, as long as version 91 seems to be deprecated, it seems that it doesn't need to be on hold anymore.

But you're the boss 😅, I'm just a random user passing by.

Bye.

[Betterbird]

Well, "we might" is conditional tense. Yes, we're at 102 now. We haven't heard any complaints about this edge case, so we're following "biggest bang for the buck" and focussing our efforts elsewhere.

[CapitainFlam]

👍
seems legit ;-)
Thanks for the reply.

[kupietools]

Hi, I'm your friendly neighborhood edge case complainant. I just tripped over this and it nearly caused such a bad catastrophe that I'm now afraid to open Betterbird at all because of how weird and unpredictible finding this kind of behavior makes the program seem.

I had to do this about 4 times before I was sure Betterbird was doing it and thought to take some screenshots.

I entered this filter: match every email older than 60 days AND not tagged "timely" AND ( (sender 1 AND subject 1) OR (sender 2 AND subject 2) OR (sender 3 AND subject 3) )

I don't know at what point it stopped saving properly, but I closed the filter editing window and immediately reopened it, and it had silently changed to this: match every email older than 60 days OR not tagged "timely" OR (sender 1 AND subject 1) OR ( (sender 2 AND subject 2) OR (sender 3 AND subject 3) ), which would have matched every single one of the 15 years worth of emails in my account.

Fortunately, at the last minute I thought to set the action to tag matched messages, not delete them, out of an abundance of caution. Not that it matters, as the filter appears to be completely broken, it looks like it has never matched anything anyway. But this is a frighteningly close call to accidentally creating a filter that deletes all my email without my even knowing it, totally different from the logic I specified.

To me this is terrifying showstopper bug that completely rules out even having a copy of Betterbird on my computer, lest I accidentally open it, because it opens up the question of what other destructive things it might be doing that I have no idea at all about. Besides which, complex filters were the primary reason I switched to Betterbird. With them not working reliably, for my own purposes there's not really any advantage to using Betterbird.

Now I'm wondering how many other filters are changed from what I entered and not relishing having to comb through them all by hand, I didn't think I needed to keep notes on what I changed when I got Betterbird. I had way higher hopes from Betterbird than tripping on a behavior this unpredictable in my first week of using it.

Sorry, I'm really not trying to disparage you folks' very obvious dedication and efforts, but the fact that this might have brought me within a hair's breadth of silently deleting 15 years' worth of emails without my even realizing it is seriously scary.

EDIT: As an experiment, disconnected from the internet (to prevent any possible data loss on the server), opened and quit Thunderbird, then opened Betterbird again. That's all I've done since the last screenshot. Now, the filter has changed again. Attempting to edit it gives this error:

It's like it knew I posted here. Either way, I'm... er... nonplussed... by filters repeatedly changing on their own.

Dismissing the error and opening the rule editing view, it has indeed changed again:

EDIT 2: This has nothing to do with editing an existing filter. It appears Betterbird fundamentally does not save filters nested more than one level deep correctly, in any case. I re-entered the entire filter by hand, without deleting any rows, closed the filters window and reopened it, and when I went back in it had once again shuffled everything around and changed the top AND term to OR. Several repeated tests confirmed this. So even entering things carefully the first time and not editing them does not avoid this bug, at least sometimes.

I thought about just editing the filter file in a text editor, but I can't make sense of the combination of backets and parentheses. I couldn't create a filter that way that Betterbird even understood, it just disabled the rule and when I tried to enable it gave me an error saying it couldn't understand it.

Respectfully, I would suggest prohibiting entering second-level nested terms at all, rather than continuing to allow them to be entered but then silently changing the logic on save without the user being alerted in any way. Just one fellow developer's opinion... unpredictably changing the logic of filter criteria without the user's knowledge strikes me as potentially dangerous. Thanks.

EDIT 3: Oops, perhaps this belongs under #62. Not sure. It's late.

[Betterbird]

Thanks for the report. It's off-topic for this ticket which is about deletion of terms from complex expressions.

BTW, opening TB on the BB rules will surely cause complete havoc, which is also stated in the "compatibility notes":
https://www.betterbird.eu/support/
Thunderbird does not recognise complex search terms defined in Betterbird.

We've created a new Issue #256 for the problem you've reported.