problem with utf8 characters making query crash

[kiwisigi]

Hi,
I am stuck with a problem where a simple query crashes when certain characters are in a utf8 table field.
This is very simple to reproduce and takes 1 minute only.
The database (fb3) attached contains 1 table with 1 record and the sql statement below fails.
The offending text in the field is: [mɒθ]

FORWADE.zip

The sql statement:

select * from qfind qf where
(qf.gno = 45901) and (qf.sfld containing '45901')

There are other expressions which cause the same error e.g.
'ʃa:bɐ
[ɕiɑu'jɑŋ] [tʂɑŋ]

I have no workaround for this as the data comes in via people entering "whatever they see fit" into a website form.

I ran all the data in my production database through a filter which filters out characters which are not:
system.character.IsLetterOrDigit
system.character.IsPunctuation
system.character.IsSeparator
system.character.IsSymbol
system.character.IsNumber
system.character.IsWhitespace -> which i replace with a blank/space character.
I was hoping to get rid of the "forbidden characters" but the ones above came through and cause a big problem.

What to do?

thanks
sigi

[asfernandes]

What OS and connection charset?

[asfernandes]

What you describe as "crashes"?

Is it an Firebird error (-string right truncation -expected length 17, actual 30) or a process crash?

[kiwisigi]

Hi Fernandes,
Yes it is and expected lenth - actual division by zero error.
Please just open the table attached and run the sql and you can see what happens.

thanks
sigi

[kiwisigi]

I use windows10 and connect with ibexpert. UTF8 is the character set.

thanks
sigi

[kiwisigi]

sorry I might have closed the issue by mistake and reopened it again.

[kiwisigi]

according to a friend of mine which has examined the issue a bit further:

The offending text in the field is: [mɒθ]
U+0252 : LATIN SMALL LETTER TURNED ALPHA
U+03B8 : GREEK SMALL LETTER THETA
But there are others too...

[asfernandes]

I do not see division by zero error when testing.

What is exactly the error you see?

Please past the error as text or image.

[kiwisigi]

I have emailed you the answer - but it does not turn up here?

Anyway my friend now made me aware that this issue has been reported already in 2016:
#5534
That looks exactly what I seem to get....

[kiwisigi]

issue: #5534 is still open - does this mean its not fixed and will not be fixed???

[asfernandes]

The bugs have the same underline causes, but they are not a single bug.

#5534 is going to be fixed sooner.

[kiwisigi]

Can I assume that if I filter out the "forbidden" characters mentioned at #5534 that the problem with the "containing" where clause is also gone?
Is this a workaround?
Will this be fixed in fb 3 or will it only be fixed in fb 4 e.g. do I have to upgrade?