PARQUET-2150: parquet-protobuf to compile on Mac M1

[steveloughran]

Force use of x86 protocol executable in builds onl macOS systems,
including those with ARM64 architectures, for which native binaries
do not exist.

Jira

• My PR addresses the following Parquet Jira issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR"
  • https://issues.apache.org/jira/browse/PARQUET-XXX
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

It can only be validated on an M1 mac. the normal CI test runs will verify the changes do not cause regressions elsewhere.

Note: this patch was based on hadoop's HADOOP-17939 patch apache/hadoop#3486 -this was needed to fix the hadoop build

Commits

• My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain Javadoc that explain what it does

[steveloughran]

fyi @dongjoon-hyun

[dongjoon-hyun]

Thank you for pinging me, @steveloughran .

[theosib-amazon]

I looked up parquet-protobuf and only found Java code. Is there some use of JNI here that I'm not finding?

[steveloughran]

its the maven protoc compiler plugin; wants to pull in something native. i have no idea why, but do know, if it is not there, you get to see a stack trace.

[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (generate-sources) on project parquet-protobuf: Error resolving artifact: com.google.protobuf:protoc:3.16.1: Could not find artifact com.google.protobuf:protoc:exe:osx-aarch_64:3.16.1 in jitpack.io (https://jitpack.io)
[ERROR] 
[ERROR] Try downloading the file manually from the project website.
[ERROR] 
[ERROR] Then, install it using the command: 
[ERROR]     mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file
[ERROR] 
[ERROR] Alternatively, if you host your own repository you can deploy the file there: 
[ERROR]     mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR] 
[ERROR]   com.google.protobuf:protoc:exe:3.16.1
[ERROR] 
[ERROR] from the specified remote repositories:
[ERROR]   jitpack.io (https://jitpack.io, releases=true, snapshots=true),
[ERROR]   apache.snapshots (https://repository.apache.org/snapshots, releases=false, snapshots=true),
[ERROR]   maven-central (https://repo1.maven.org/maven2, releases=true, snapshots=false)
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :parquet-protobuf

[steveloughran]

ok, somehow I've broken the default classifier logic, which is of course critical.

downgrading this to a draft until it's ready to be reviewed

Error:  Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (generate-sources) on project parquet-protobuf: Error resolving artifact: com.google.protobuf:protoc:3.16.1:exe:${os.detected.classifier}: Could not find artifact com.google.protobuf:protoc:exe:${os.detected.classifier}:3.16.1 in jitpack.io (https://jitpack.io/)
Error:  
Error:  Try downloading the file manually from the project website.
Error:  
Error:  Then, install it using the command: 
Error:      mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=${os.detected.classifier} -Dpackaging=exe -Dfile=/path/to/file
Error:  
Error:  Alternatively, if you host your own repository you can deploy the file there: 
Error:      mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=${os.detected.classifier} -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

[steveloughran]

tested lasted patch on

• macbook m1
• macbook x86
• ec2 x86

i would set up my raspberry pi for a build too -I just fear having to build the thrift compiler

[sunchao]

Oops @steveloughran just found out this PR. I opened #973 for the same purpose

[shangxinli]

@sunchao I see your change is to upgrade the protobuf version. Is that required to solve this problem, which I don't see in this PR.

[sunchao]

@shangxinli my approach is different from @steveloughran 's one. Since newer version of protobuf already provides M1 artifacts, upgrade will solve the issue

[steveloughran]

this patch is based on Dongjoon;s one for hadoop, tells maven to use the x86 artifact on macbook m1 builds.

the sunchao one switches to a version of protobuf with a genuine mac m1 artifacts, a version which should also include some CVE fixes.

[shangxinli]

@steveloughran Thanks for the explanation! Do you have concerns if we use PR-973? It seems we can rely on proto-buf itself to solve the issue.

@sunchao please add @steveloughran as co-author for your PR.

[steveloughran]

oh, upgrading is better!

[steveloughran]

resolved by #973