[Bug] [Maxcompute] Failed to parse some maxcompute type #3894

stdnt-xiao · 2023-01-07T03:32:00Z

Purpose of this pull request

I have tested most data types. eg

set odps.sql.hive.compatible=true;

DROP TABLE IF EXISTS fake_source;

CREATE TABLE IF NOT EXISTS fake_source(c1 TINYINT,c2 SMALLINT,c3 INT,c4 BIGINT,c5 FLOAT ,c6 DOUBLE
,c7 VARCHAR(10),c8 CHAR(10),c9 STRING,c10 DATE,c11 DATETIME ,c12 TIMESTAMP ,c13 BOOLEAN,c14 BINARY
,c15 MAP<STRING,STRING>,c16 ARRAY,c17 STRUCT<s1:STRING,s2:INT,s3:ARRAY>);

INSERT INTO fake_source(c1, c2, c3, c4, c5, c6, c7,c8,c9,c10,c11,c12,c13,c14,c15,c16,c17) VALUES (
CAST(-128 AS TINYINT ),CAST(-32768 AS SMALLINT ) ,0,10000000000000,0.01,0.0000000000000001
,CAST("varchar" as VARCHAR(10)),CAST("char" as CHAR(10)),"hello0",CAST("2022-12-31" as DATE )
,CAST("2022-12-31 23:59:59" as DATETIME ),CAST("2022-12-31 23:59:59.999" as TIMESTAMP ),FALSE,CAST("bytes" AS BINARY )
,MAP("int",1,"str","hello"),ARRAY("11","22"),named_struct("s1","s1","s2",100,"s3",array(1.1, 2.2)));

SELECT * FROM fake_source;

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

stdnt-xiao · 2023-01-10T13:43:31Z

@TaoZex
Can you give me some reminders? thx.

TyrantLucifer · 2023-01-11T09:26:16Z

...nnectors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleArrayTypeInfo.java

+
+    SimpleArrayTypeInfo(TypeInfo typeInfo) {
+        if (typeInfo == null) {
+            throw new IllegalArgumentException("Invalid element type.");


throw new MaxcomputeConnectorException(CommonErrorCode.UNSUPPORTED_DATA_TYPE)

TyrantLucifer · 2023-01-11T09:26:38Z

...connectors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleMapTypeInfo.java

+            this.keyType = keyType;
+            this.valueType = valueType;
+        } else {
+            throw new IllegalArgumentException("Invalid key or value type for map.");


Same as above.

TyrantLucifer · 2023-01-11T09:27:01Z

...nectors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleStructTypeInfo.java

+    private void validateParameters(List<String> names, List<TypeInfo> typeInfos) {
+        if (names != null && typeInfos != null && !names.isEmpty() && !typeInfos.isEmpty()) {
+            if (names.size() != typeInfos.size()) {
+                throw new IllegalArgumentException("The amount of field names must be equal to the amount of field types.");


Same as above.

TyrantLucifer · 2023-01-11T09:27:09Z

...nectors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleStructTypeInfo.java

+                throw new IllegalArgumentException("The amount of field names must be equal to the amount of field types.");
+            }
+        } else {
+            throw new IllegalArgumentException("Invalid name or element type for struct.");


Same as above.

…TED_DATA_TYPE)

stdnt-xiao · 2023-01-13T00:59:52Z

@TyrantLucifer
I have fixed the problem you pointed out. thx

Hisoka-X · 2023-01-16T07:35:02Z

Can you help to add MaxCompute E2E, if so will be very helpful.

stdnt-xiao · 2023-01-17T00:29:55Z

Can you help to add MaxCompute E2E, if so will be very helpful.

@Hisoka-X
I'm happy to complete it, but maxcompute is a commercial product of Alibaba Cloud. I can't build an instance through docker image.

Hisoka-X · 2023-01-17T02:03:38Z

Can you help to add MaxCompute E2E, if so will be very helpful.

@Hisoka-X I'm happy to complete it, but maxcompute is a commercial product of Alibaba Cloud. I can't build an instance through docker image.

Got it.

CalvinKirs · 2023-01-17T07:43:35Z

Can you help to add MaxCompute E2E, if so will be very helpful.↳

@Hisoka-X I'm happy to complete it, but maxcompute is a commercial product of Alibaba Cloud. I can't build an instance through docker image.↳

It would be even better if you can provide the corresponding test code, you can set it to disable, that is, it will not be executed by default, but other developers, they can perform simple configurations according to their own environment to execute IT

stdnt-xiao · 2023-01-18T07:06:42Z

Can you help to add MaxCompute E2E, if so will be very helpful.↳

@Hisoka-X I'm happy to complete it, but maxcompute is a commercial product of Alibaba Cloud. I can't build an instance through docker image.↳

It would be even better if you can provide the corresponding test code, you can set it to disable, that is, it will not be executed by default, but other developers, they can perform simple configurations according to their own environment to execute IT

OK, I will supplement.

stdnt-xiao · 2023-01-19T07:19:41Z

@Hisoka-X @CalvinKirs
I have added MaxCompute E2E. like connector-datahub-e2e

CodingGPT · 2023-02-15T15:59:14Z

I tested the code using a tableSchema like "id BIGINT, shopname STRING, start_date DATE, price DOUBLE, create_time DATETIME, update_timestamp TIMESTAMP, f1 tinyint, f2 smallint, f3 int, f4 float, f5 decimal(38,18), f6 char(20), f7 varchar(30), f8 boolean" and sink to mysql , mostly get the right result.

…TED_DATA_TYPE)

TyrantLucifer

Overall LGTM,but you should offer the test snapshot images to verify this pull request is valid. Thank you for your contribution of SeaTunnel.

# Conflicts: # seatunnel-connectors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleArrayTypeInfo.java # seatunnel-connectors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleMapTypeInfo.java # seatunnel-connectors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleStructTypeInfo.java # seatunnel-connectors-v2/connector-maxcompute/src/main/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/sink/MaxcomputeWriter.java # seatunnel-connectors-v2/connector-maxcompute/src/main/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/util/MaxcomputeTypeMapper.java # seatunnel-connectors-v2/connector-maxcompute/src/test/java/BasicTypeToOdpsTypeTest.java # seatunnel-e2e/seatunnel-connector-v2-e2e/connector-maxcompute-e2e/pom.xml # seatunnel-e2e/seatunnel-connector-v2-e2e/connector-maxcompute-e2e/src/test/java/org/apache/seatunnel/e2e/connector/maxcompute/MaxcomputeIT.java # seatunnel-e2e/seatunnel-connector-v2-e2e/pom.xml

stdnt-xiao · 2023-03-04T10:57:03Z

Overall LGTM,but you should offer the test snapshot images to verify this pull request is valid. Thank you for your contribution of SeaTunnel.

Maxcompute is a commercial product of Alibaba Cloud, so I can only provide screenshots.

hailin0

please update

https://github.com/apache/incubator-seatunnel/blob/dev/release-note.md
https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/source/Maxcompute.md
https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/sink/Maxcompute.md

Hisoka-X

LGTM

Dannila · 2023-07-26T09:36:05Z

Please help me to see if there is something wrong with my configuration file, the configuration file is as follows:

{
  env {
    execution.parallelism = 1
    job.mode = "BATCH"
  }
  source {
    Maxcompute {
      parallelism = 3
      accessId = "LxxxxxxxxxxxxxxxxxxxxxxxK"
      accesskey = "9xxxxxxxxxxxxxxxxxxxxxxxj"
      endpoint = "http://service.cn-shanghai.maxcompute.aliyun.com/api"
      project = "maxcompute_test_1"
      table_name = "ods_xxxxxxxxx_month_dailyupdate"
    }
  }
  sink {
    Jdbc {
          url = "jdbc:mysql://172.xx.x.xxx:xxxx"
          driver = "com.mysql.cj.jdbc.Driver"
          connection_check_timeout_sec = 100
          user = "root"
          password = "xxxxxxx"
          query = "insert into `ods`.`ods_repo_month_dailyupdate_maxc` (repo_id, repo_name, stars, commits, pushes, pr_creators, pr_reviews, pr_reviewers, total_issues, forks, month) values (?,?,?,?,?,?,?,?,?,?,?,?);"
      }
  }
}

My execution command is as follows：

${SEATUNNEL_HOME}/bin/start-seatunnel-flink-15-connector-v2.sh --config /opt/apache-seatunnel-2.3.2/script/ods_repo.conf -e run

But the task error log is as follows：

Caused by: org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Cannot initialize resource provider.
   at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.initialize(ActiveResourceManager.java:158) ~[flink-dist-1.15.3.jar:1.15.3]
   at org.apache.flink.runtime.resourcemanager.ResourceManager.startResourceManagerServices(ResourceManager.java:241) ~[flink-dist-1.15.3.jar:1.15.3]
   at org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:218) ~[flink-dist-1.15.3.jar:1.15.3]
   ... 24 more
Caused by: org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Could not start resource manager client.
   at org.apache.flink.yarn.YarnResourceManagerDriver.initializeInternal(YarnResourceManagerDriver.java:191) ~[flink-dist-1.15.3.jar:1.15.3]
   at org.apache.flink.runtime.resourcemanager.active.AbstractResourceManagerDriver.initialize(AbstractResourceManagerDriver.java:81) ~[flink-dist-1.15.3.jar:1.15.3]
   at org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.initialize(ActiveResourceManager.java:156) ~[flink-dist-1.15.3.jar:1.15.3]
   at org.apache.flink.runtime.resourcemanager.ResourceManager.startResourceManagerServices(ResourceManager.java:241) ~[flink-dist-1.15.3.jar:1.15.3]
   at org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:218) ~[flink-dist-1.15.3.jar:1.15.3]
   ... 24 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.reflect.InvocationTargetException
   at org.apache.hadoop.yarn.factories.impl.pb.RpcClientFactoryPBImpl.getClient(RpcClientFactoryPBImpl.java:81) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.2]
...

[Bug] [Maxcompute] Failed to parse some maxcompute type apache#3893

cc98993

stdnt-xiao changed the title ~~[Bug] [Maxcompute] Failed to parse some maxcompute type #3893~~ [Bug] [Maxcompute] Failed to parse some maxcompute type Jan 7, 2023

stdnt-xiao closed this Jan 7, 2023

stdnt-xiao reopened this Jan 7, 2023

stdnt-xiao mentioned this pull request Jan 10, 2023

[Improve][Maxcompute] Throws exception with unclear message. #3903

Closed

5 tasks

TyrantLucifer reviewed Jan 11, 2023

View reviewed changes

stdnt-xiao added 2 commits January 13, 2023 08:48

[Fix] throw new MaxcomputeConnectorException(CommonErrorCode.UNSUPPOR…

f263a65

…TED_DATA_TYPE)

[fix] check code style

5f24948

stdnt-xiao requested a review from TyrantLucifer January 14, 2023 02:52

Hisoka-X added bug connectors-v2 labels Jan 16, 2023

stdnt-xiao added 2 commits January 19, 2023 14:15

Merge branch 'dev' into dev-3893

1e12092

[Improve] add connector-maxcompute-e2e

1f4bcc4

stdnt-xiao added 2 commits January 19, 2023 15:22

[Fix] pom.xml don't have a valid license header

8277a30

[Fix] rename directory

fd4dc68

stdnt-xiao mentioned this pull request Feb 1, 2023

[Bugfix][Connector-V2][maxcompute] spark engine support date type #4009

Closed

5 tasks

stdnt-xiao added 5 commits February 16, 2023 00:29

[Bug] [Maxcompute] Failed to parse some maxcompute type apache#3893

a8d593f

[Fix] throw new MaxcomputeConnectorException(CommonErrorCode.UNSUPPOR…

7263e69

…TED_DATA_TYPE)

[fix] check code style

20deeae

[Improve] add connector-maxcompute-e2e

63520fb

[Fix] pom.xml don't have a valid license header

61366de

stdnt-xiao and others added 2 commits February 16, 2023 00:30

[Fix] rename directory

4cc87e7

[Hotfix][MaxCompute] Rebase dev and reformat with spotless

6e9d3c8

TyrantLucifer force-pushed the dev-3893 branch from fd4dc68 to 6e9d3c8 Compare February 15, 2023 16:32

TyrantLucifer previously approved these changes Mar 4, 2023

View reviewed changes

github-actions bot added approved reviewed labels Mar 4, 2023

stdnt-xiao and others added 3 commits March 4, 2023 17:13

Merge branch 'apache:dev' into dev-3893

4b574b7

merge dev laster

df676d7

hailin0 reviewed Mar 4, 2023

View reviewed changes

Update release-note.md

3986e5d

stdnt-xiao dismissed TyrantLucifer’s stale review via 3986e5d March 4, 2023 12:06

github-actions bot removed approved reviewed labels Mar 4, 2023

hailin0 approved these changes Mar 4, 2023

View reviewed changes

github-actions bot added approved reviewed labels Mar 4, 2023

Hisoka-X approved these changes Mar 4, 2023

View reviewed changes

Hisoka-X merged commit 642901f into apache:dev Mar 4, 2023

stdnt-xiao deleted the dev-3893 branch March 4, 2023 12:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] [Maxcompute] Failed to parse some maxcompute type #3894

[Bug] [Maxcompute] Failed to parse some maxcompute type #3894

stdnt-xiao commented Jan 7, 2023 •

edited by Hisoka-X

Loading

stdnt-xiao commented Jan 10, 2023

TyrantLucifer Jan 11, 2023

TyrantLucifer Jan 11, 2023

TyrantLucifer Jan 11, 2023

TyrantLucifer Jan 11, 2023

stdnt-xiao commented Jan 13, 2023

Hisoka-X commented Jan 16, 2023

stdnt-xiao commented Jan 17, 2023 •

edited

Loading

Hisoka-X commented Jan 17, 2023

CalvinKirs commented Jan 17, 2023

stdnt-xiao commented Jan 18, 2023

stdnt-xiao commented Jan 19, 2023

CodingGPT commented Feb 15, 2023

TyrantLucifer left a comment

stdnt-xiao commented Mar 4, 2023

hailin0 left a comment

Hisoka-X left a comment

Dannila commented Jul 26, 2023

[Bug] [Maxcompute] Failed to parse some maxcompute type #3894

[Bug] [Maxcompute] Failed to parse some maxcompute type #3894

Conversation

stdnt-xiao commented Jan 7, 2023 • edited by Hisoka-X Loading

Purpose of this pull request

Check list

stdnt-xiao commented Jan 10, 2023

TyrantLucifer Jan 11, 2023

Choose a reason for hiding this comment

TyrantLucifer Jan 11, 2023

Choose a reason for hiding this comment

TyrantLucifer Jan 11, 2023

Choose a reason for hiding this comment

TyrantLucifer Jan 11, 2023

Choose a reason for hiding this comment

stdnt-xiao commented Jan 13, 2023

Hisoka-X commented Jan 16, 2023

stdnt-xiao commented Jan 17, 2023 • edited Loading

Hisoka-X commented Jan 17, 2023

CalvinKirs commented Jan 17, 2023

stdnt-xiao commented Jan 18, 2023

stdnt-xiao commented Jan 19, 2023

CodingGPT commented Feb 15, 2023

TyrantLucifer left a comment

Choose a reason for hiding this comment

stdnt-xiao commented Mar 4, 2023

hailin0 left a comment

Choose a reason for hiding this comment

Hisoka-X left a comment

Choose a reason for hiding this comment

Dannila commented Jul 26, 2023

stdnt-xiao commented Jan 7, 2023 •

edited by Hisoka-X

Loading

stdnt-xiao commented Jan 17, 2023 •

edited

Loading