[Feature][Connector-V2][OceanBase] Support vector types on OceanBase

[xxsc0529]

Purpose of this pull request

Added support for vectors in Oceanbase Close #7290

Does this PR introduce any user-facing change?

yes

How was this patch tested?

When creating test data in milvus, the database is default, the collection is created as simple_examole, and the data sample is as shown in the following figure

The test script is as follows
env {
job.mode = "BATCH"
}

source {
Milvus {
url = "http://localhost:19530"
token = "root:Milvus"
database = "default"
collection="simple_example"
}
}

transform {
}

sink {
jdbc {
url = "jdbc:oceanbase://localhost:2881/test"
driver = "com.oceanbase.jdbc.Driver"
user = "root"
password = ""
generate_sink_sql =true
compatible_mode="mysql"
database = "test"
table = "simple_example"
}
}
Since oceanbase-client does not support vector parsing, oceanbase catalog does not support automatic creation of vector tables
The DDL of Oceanbase is as follows:
CREATE TABLE IF NOT EXISTS simple_example
(
book_id int NOT NULL,
book_intro vector(4) DEFAULT NULL,
book_title varchar(64) DEFAULT NULL,
primary key (book_id)
);

This mission was successful

Check list

• If any new Jar binary package adding in your PR, please add License Notice according
  New License Guide
• If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
• If you are contributing the connector code, please check that the following files are updated:
  1. Update change log that in connector document. For more details you can refer to connector-v2
  2. Update plugin-mapping.properties and add new connector information in it
  3. Update the pom file of seatunnel-dist
• Update the release-note.

[Hisoka-X]

Could you add a test case for insert/read vector into/from oceanbase?

[xxsc0529]

Could you add a test case for insert/read vector into/from oceanbase?

I added the test case and the test on giothub was successful

Because the milvus container and oceanbase container occupy too much memory after startup, @disable annotations are added

[Hisoka-X]

Thanks @xxsc0529 ! How about waiting #7401 be merged to replace milvus in test case?

[xxsc0529]

Thanks @xxsc0529 ! How about waiting #7401 be merged to replace milvus in test case?

ok

[xxsc0529]

Thanks @xxsc0529 ! How about waiting #7401 be merged to replace milvus in test case?

@Hisoka-X this #7401 is merged

[Hisoka-X]

Please merge from dev then use fake source to produce vector data to write into oceanbase

[xxsc0529]

Please merge from dev then use fake source to produce vector data to write into oceanbase
The test case of fakesource has been added, and the result is as follows

[xxsc0529]

Please merge from dev then use fake source to produce vector data to write into oceanbase
What should I do if I find that the connection fails to execute the workflow?
The link is below
https://paimon.apache.org/docs/0.6/maintenance/configurations/#coreoptions
location
./docs/en/connector-v2/sink/Paimon.md

@Hisoka-X

[Hisoka-X]

Please merge from dev then use fake source to produce vector data to write into oceanbase
What should I do if I find that the connection fails to execute the workflow?
The link is below
https://paimon.apache.org/docs/0.6/maintenance/configurations/#coreoptions
location
./docs/en/connector-v2/sink/Paimon.md

@Hisoka-X

try to fix it in #7435

[Hisoka-X]

Please merge from dev. Thanks

[xxsc0529]

Please merge from dev. Thanks

done,

Did you change the logic of mysqlcdc? There is an error here @Hisoka-X

[Hisoka-X]

Did you change the logic of mysqlcdc? There is an error here @Hisoka-X

No, maybe it's unstable. Please retry failed CI.

[xxsc0529]

Did you change the logic of mysqlcdc? There is an error here @Hisoka-X

No, maybe it's unstable. Please retry failed CI.

ok,done, @Hisoka-X

[Hisoka-X]

Please remove disabled on fake to oceanbase case.

[xxsc0529]

At present, Oceanbase's vector image will not be closed, resulting in a process still running, and this image problem will be dealt with in the future, but the function is okay。seatunne's executeJob logic has an assertion to determine whether there is an image running, and after testing, replacing it with a mysql image or other versions of the image will not have this error, and the current ob vector image has this problem, so add @disable without modifying the original test logic on the premise。

[Hisoka-X]

Oceanbase's vector image will not be closed, resulting in a process still running

Do you know the reason? And is there any problem if we don't solve this problem and enable the test case at the same time?

[xxsc0529]

Please remove disabled on fake to oceanbase case.

Subsequent PR will be submitted, the OB-related testcontainers will be modified, and the test container will be replaced to the latest version @Hisoka-X What do you think

[xxsc0529]

Oceanbase's vector image will not be closed, resulting in a process still running

Do you know the reason? And is there any problem if we don't solve this problem and enable the test case at the same time?

The problem of the ob vector container, I am sure that there is no problem with the test case, and the synchronous results can be printed out at the moment, but the test container still has threads running and cannot be closed

[Hisoka-X]

ok.

[Hisoka-X]

At present, Oceanbase's vector image will not be closed, resulting in a process still running, and this image problem will be dealt with in the future, but the function is okay。seatunne's executeJob logic has an assertion to determine whether there is an image running, and after testing, replacing it with a mysql image or other versions of the image will not have this error, and the current ob vector image has this problem, so add @disable without modifying the original test logic on the premise。

This is the assert logic provided by seatunnel to ensure that there are no abnormal threads in the engine. I believe this test case did not pass our assert logic. The JNA cleaner thread come from oceanbase client. This is a bug come from jna tool. Fixed by java-native-access/jna@e8182b2 in version 5.14.0
So we should do this:

1. open test case and add JNA cleaner thread into

   seatunnel/seatunnel-e2e/seatunnel-e2e-common/src/test/java/org/apache/seatunnel/e2e/common/container/seatunnel/SeaTunnelContainer.java

   Line 407 in 570bbb3

   private boolean isIssueWeAlreadyKnow(String threadName) {

2. waiting oceanbase client update jna version to 5.14.0 then remove it from isIssueWeAlreadyKnow

[Hisoka-X]

cc @whhe as well.

[Hisoka-X]

LGTM. Thanks @xxsc0529 !