Skip to content

[HUDI-4972] Fixes to make unit tests work on m1 #6751

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nsivabalan merged 7 commits into from
Oct 7, 2022
Merged

[HUDI-4972] Fixes to make unit tests work on m1 #6751

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d01239a
fixes to make unit tests on m1
Sep 22, 2022
1cbb28c
also added an m1 compatible version for rocksdb jni
Sep 23, 2022
afc44f2
Merge branch 'apache:master' into m1_unit_test_compatibility
jonvex Sep 28, 2022
c7a1d37
Merge branch 'apache:master' into m1_unit_test_compatibility
jonvex Sep 29, 2022
b8ffeeb
got tests working
Sep 29, 2022
ea52ad8
removed the exclusions that we had previously
Sep 29, 2022
287c52c
reverted the sql test
Sep 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions hudi-examples/hudi-examples-java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,12 @@
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<scope>provided</scope>
</dependency>

<!-- Junit dependencies -->
<dependency>
<groupId>org.junit.jupiter</groupId>
Expand Down
5 changes: 5 additions & 0 deletions hudi-timeline-service/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,11 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<scope>provided</scope>
</dependency>

<!-- Hoodie - Test -->
<dependency>
Expand Down
34 changes: 33 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@
<flink.clients.artifactId>flink-clients</flink.clients.artifactId>
<flink.connector.kafka.artifactId>flink-connector-kafka</flink.connector.kafka.artifactId>
<flink.hadoop.compatibility.artifactId>flink-hadoop-compatibility_2.12</flink.hadoop.compatibility.artifactId>
<rocksdbjni.version>5.17.2</rocksdbjni.version>
<spark31.version>3.1.3</spark31.version>
<spark32.version>3.2.2</spark32.version>
<spark33.version>3.3.0</spark33.version>
Expand Down Expand Up @@ -205,6 +206,7 @@
<dynamodb-local.endpoint>http://localhost:${dynamodb-local.port}</dynamodb-local.endpoint>
<springboot.version>2.7.3</springboot.version>
<spring.shell.version>2.1.1</spring.shell.version>
<snappy.version>1.1.8.3</snappy.version>
</properties>

<scm>
Expand Down Expand Up @@ -389,12 +391,16 @@
<exclude>org.apache.hbase:hbase-common:*</exclude>
<exclude>org.apache.hbase:hbase-client:*</exclude>
<exclude>org.apache.hbase:hbase-server:*</exclude>
<!--To upgrade snappy because pre 1.1.8.2 does not work on m1 mac-->
<exclude>org.xerial.snappy:snappy-java:*</exclude>
</excludes>
<includes>
<include>org.slf4j:slf4j-simple:*:*:test</include>
<include>org.apache.hbase:hbase-common:${hbase.version}</include>
<include>org.apache.hbase:hbase-client:${hbase.version}</include>
<include>org.apache.hbase:hbase-server:${hbase.version}</include>
<!--To upgrade snappy because pre 1.1.8.2 does not work on m1 mac-->
<include>org.xerial.snappy:snappy-java:${snappy.version}</include>
</includes>
</bannedDependencies>
</rules>
Expand Down Expand Up @@ -739,9 +745,22 @@
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>${avro.version}</version>
<exclusions>
<exclusion>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
</exclusion>
</exclusions>
<scope>provided</scope>
</dependency>

<!-- Snappy -->
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<version>${snappy.version}</version>
</dependency>

<!-- caffeine -->
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
Expand Down Expand Up @@ -909,7 +928,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
<version>5.17.2</version>
<version>${rocksdbjni.version}</version>
</dependency>

<!-- Httpcomponents -->
Expand Down Expand Up @@ -1997,6 +2016,19 @@
</property>
</activation>
</profile>
<profile>
<id>m1-mac</id>
<properties>
<spark2.version>2.4.8</spark2.version>
<rocksdbjni.version>6.29.4.1</rocksdbjni.version>
</properties>
<activation>
<os>
<family>mac</family>
<arch>aarch64</arch>
</os>
</activation>
</profile>

<!-- "spark3" is an alias for "spark3.3" -->
<!-- NOTE: This profile is deprecated and soon will be removed -->
Expand Down