Skip to content

[SPARK-48284][SQL] Fix UTF8String indexOf behaviour for empty string search #46581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
uros-db wants to merge 9 commits into from
Closed

[SPARK-48284][SQL] Fix UTF8String indexOf behaviour for empty string search #46581

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
470a9c9
Initial commit
uros-db May 14, 2024
be2c09a
Merge branch 'apache:master' into utf8string-indexof
uros-db May 15, 2024
2bdfb1e
Merge branch 'apache:master' into utf8string-indexof
uros-db May 28, 2024
31d93da
Merge branch 'master' into utf8string-indexof
uros-db May 30, 2024
e3ad4da
Update UTF8StringSuite.java
uros-db May 30, 2024
9e81538
Fix uses
uros-db May 31, 2024
78b981a
Update CollationAwareUTF8String.java
uros-db Oct 22, 2024
ed809fb
Merge branch 'master' into utf8string-indexof
uros-db Oct 22, 2024
78a78ee
update
uros-db Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -345,14 +345,14 @@ public static int findInSet(final UTF8String match, final UTF8String set, int co
*/
public static int lowercaseIndexOf(final UTF8String target, final UTF8String pattern,
final int start) {
if (pattern.numChars() == 0) return target.indexOfEmpty(start);
if (pattern.numChars() == 0) return target.indexOfEmpty(target, start);
return lowercaseFind(target, pattern.toLowerCase(), start);
}

public static int indexOf(final UTF8String target, final UTF8String pattern,
final int start, final int collationId) {
if (pattern.numBytes() == 0) {
return target.indexOfEmpty(start);
return target.indexOfEmpty(target, start);
}

StringSearch stringSearch = CollationFactory.getStringSearch(target, pattern, collationId);
Expand Down
6 changes: 3 additions & 3 deletions common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -780,8 +780,8 @@ public UTF8String repeat(int times) {
* @param start the start position of the current string for searching
* @return the position of the first occurrence of the empty substr (now, always 0)
*/
public int indexOfEmpty(int start) {
return 0; // TODO: Fix this behaviour (SPARK-48284)
public int indexOfEmpty(UTF8String v, int start) {
return start >= numChars() ? 0 : start;
}

/**
Expand All @@ -794,7 +794,7 @@ public int indexOfEmpty(int start) {
*/
public int indexOf(UTF8String v, int start) {
if (v.numBytes() == 0) {
return indexOfEmpty(start);
return indexOfEmpty(v, start);
}

// locate to the start position.
Expand Down
2 changes: 2 additions & 0 deletions common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,8 @@ public void indexOf() {
assertEquals(0, EMPTY_UTF8.indexOf(EMPTY_UTF8, 0));
assertEquals(-1, EMPTY_UTF8.indexOf(fromString("l"), 0));
assertEquals(0, fromString("hello").indexOf(EMPTY_UTF8, 0));
assertEquals(2, fromString("hello").indexOf(EMPTY_UTF8, 2));
assertEquals(0, fromString("hello").indexOf(EMPTY_UTF8, 9));
assertEquals(2, fromString("hello").indexOf(fromString("l"), 0));
assertEquals(3, fromString("hello").indexOf(fromString("l"), 3));
assertEquals(-1, fromString("hello").indexOf(fromString("a"), 0));
Expand Down