Skip to content

HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes #1494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vinayakumarb merged 4 commits into from
Sep 23, 2019
Merged

HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes #1494

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3f3f7a6
HADOOP-16558. [pb-upgrade] [COMMON] Use protobuf-maven-plugin to gene…
vinayakumarb Sep 12, 2019
f72cfca
HADOOP-16558. [pb-upgrade] [HDFS] Use protobuf-maven-plugin to genera…
vinayakumarb Sep 12, 2019
ef02172
HADOOP-16558. [pb-upgrade] [COMMON] Use protobuf-maven-plugin to gene…
vinayakumarb Sep 22, 2019
f5ca19c
Changed the execution name fot test
vinayakumarb Sep 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 14 additions & 52 deletions hadoop-common-project/hadoop-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,20 @@
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<executions>
<execution>
<id>src-compile-protoc</id>
<configuration><skip>false</skip></configuration>
</execution>
<execution>
<id>test-compile-protoc</id>
<configuration><skip>false</skip></configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-maven-plugins</artifactId>
Expand All @@ -400,58 +414,6 @@
</source>
</configuration>
</execution>
<execution>
<id>compile-protoc</id>
<goals>
<goal>protoc</goal>
</goals>
<configuration>
<protocVersion>${protobuf.version}</protocVersion>
<protocCommand>${protoc.path}</protocCommand>
<imports>
<param>${basedir}/src/main/proto</param>
</imports>
<source>
<directory>${basedir}/src/main/proto</directory>
<includes>
<include>HAServiceProtocol.proto</include>
<include>IpcConnectionContext.proto</include>
<include>ProtocolInfo.proto</include>
<include>RpcHeader.proto</include>
<include>ZKFCProtocol.proto</include>
<include>ProtobufRpcEngine.proto</include>
<include>Security.proto</include>
<include>GetUserMappingsProtocol.proto</include>
<include>TraceAdmin.proto</include>
<include>RefreshAuthorizationPolicyProtocol.proto</include>
<include>RefreshUserMappingsProtocol.proto</include>
<include>RefreshCallQueueProtocol.proto</include>
<include>GenericRefreshProtocol.proto</include>
<include>FSProtos.proto</include>
</includes>
</source>
</configuration>
</execution>
<execution>
<id>compile-test-protoc</id>
<goals>
<goal>test-protoc</goal>
</goals>
<configuration>
<protocVersion>${protobuf.version}</protocVersion>
<protocCommand>${protoc.path}</protocCommand>
<imports>
<param>${basedir}/src/test/proto</param>
</imports>
<source>
<directory>${basedir}/src/test/proto</directory>
<includes>
<include>test.proto</include>
<include>test_rpc_service.proto</include>
</includes>
</source>
</configuration>
</execution>
<execution>
<id>resource-gz</id>
<phase>generate-resources</phase>
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/FSProtos.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.fs";
option java_outer_classname = "FSProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/GenericRefreshProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.ipc.proto";
option java_outer_classname = "GenericRefreshProtocolProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/GetUserMappingsProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.tools.proto";
option java_outer_classname = "GetUserMappingsProtocolProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/HAServiceProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.ha.proto";
option java_outer_classname = "HAServiceProtocolProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/IpcConnectionContext.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.ipc.protobuf";
option java_outer_classname = "IpcConnectionContextProtos";
option java_generate_equals_and_hash = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/ProtobufRpcEngine.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
/**
* These are the messages used by Hadoop RPC for the Rpc Engine Protocol Buffer
* to marshal the request and response in the RPC layer.
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/ProtocolInfo.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.ipc.protobuf";
option java_outer_classname = "ProtocolInfoProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/RefreshAuthorizationPolicyProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.security.proto";
option java_outer_classname = "RefreshAuthorizationPolicyProtocolProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/RefreshCallQueueProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.ipc.proto";
option java_outer_classname = "RefreshCallQueueProtocolProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/RefreshUserMappingsProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.security.proto";
option java_outer_classname = "RefreshUserMappingsProtocolProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.ipc.protobuf";
option java_outer_classname = "RpcHeaderProtos";
option java_generate_equals_and_hash = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/Security.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.security.proto";
option java_outer_classname = "SecurityProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/TraceAdmin.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.tracing";
option java_outer_classname = "TraceAdminPB";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/main/proto/ZKFCProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.ha.proto";
option java_outer_classname = "ZKFCProtocolProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-common-project/hadoop-common/src/test/proto/test.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/

syntax = "proto2";
option java_package = "org.apache.hadoop.ipc.protobuf";
option java_outer_classname = "TestProtos";
option java_generate_equals_and_hash = true;
Expand Down
1 change: 1 addition & 0 deletions hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
syntax = "proto2";
option java_package = "org.apache.hadoop.ipc.protobuf";
option java_outer_classname = "TestRpcServiceProtos";
option java_generic_services = true;
Expand Down
36 changes: 9 additions & 27 deletions hadoop-hdfs-project/hadoop-hdfs-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,36 +131,18 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
</configuration>
</plugin>
<plugin>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-maven-plugins</artifactId>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<executions>
<execution>
<id>compile-protoc</id>
<goals>
<goal>protoc</goal>
</goals>
<id>src-compile-protoc</id>
<configuration>
<protocVersion>${protobuf.version}</protocVersion>
<protocCommand>${protoc.path}</protocCommand>
<imports>
<param>${basedir}/../../hadoop-common-project/hadoop-common/src/main/proto</param>
<param>${basedir}/src/main/proto</param>
</imports>
<source>
<directory>${basedir}/src/main/proto</directory>
<includes>
<include>ClientDatanodeProtocol.proto</include>
<include>ClientNamenodeProtocol.proto</include>
<include>acl.proto</include>
<include>xattr.proto</include>
<include>datatransfer.proto</include>
<include>hdfs.proto</include>
<include>encryption.proto</include>
<include>inotify.proto</include>
<include>erasurecoding.proto</include>
<include>ReconfigurationProtocol.proto</include>
</includes>
</source>
<skip>false</skip>
<additionalProtoPathElements>
<additionalProtoPathElement>
${basedir}/../../hadoop-common-project/hadoop-common/src/main/proto
Copy link
Contributor

@Apache9 Apache9 Sep 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we also attach the proto along in the jar, we do not need to add this configuration. This may change the behavior but I think it is fine? Of course, keep the old way is also fine, not a big problem. Just want to make the pom simpler. As in the old way, you need make sure that we also depend on hadoop-common in the dependencies section, otherwise the protoc generating is fine but there will be compile error...

Copy link
Contributor Author

@vinayakumarb vinayakumarb Sep 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I agree that keeping proto files in jar may be fine for now.
It would be better to specify explicitly which proto files are required for successfull proto generation. So I preferred to NOT to include in jars.

As in the old way, you need make sure that we also depend on hadoop-common in the dependencies section, otherwise the protoc generating is fine but there will be compile error...

hadoop-common in dependency section is anyway required, irrespective of whether proto files are part of jar or not.

</additionalProtoPathElement>
</additionalProtoPathElements>
</configuration>
</execution>
</executions>
Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientDatanodeProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax="proto2";
// This file contains protocol buffers that are used throughout HDFS -- i.e.
// by the client, server, and data transfer protocols.

Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax="proto2";
option java_package = "org.apache.hadoop.hdfs.protocol.proto";
option java_outer_classname = "ClientNamenodeProtocolProtos";
option java_generic_services = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ReconfigurationProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/

syntax="proto2";
// This file contains protocol buffers that are used to reconfigure NameNode
// and DataNode by HDFS admin.

Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/acl.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/

syntax="proto2";
option java_package = "org.apache.hadoop.hdfs.protocol.proto";
option java_outer_classname = "AclProtos";
option java_generate_equals_and_hash = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax="proto2";
// This file contains protocol buffers that are used to transfer data
// to and from the datanode, as well as between datanodes.

Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/encryption.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax="proto2";
// This file contains protocol buffers that are used throughout HDFS -- i.e.
// by the client, server, and data transfer protocols.

Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/erasurecoding.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
syntax="proto2";
option java_package = "org.apache.hadoop.hdfs.protocol.proto";
option java_outer_classname = "ErasureCodingProtos";
option java_generate_equals_and_hash = true;
Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax="proto2";
// This file contains protocol buffers that are used throughout HDFS -- i.e.
// by the client, server, and data transfer protocols.

Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/inotify.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
* Please see http://wiki.apache.org/hadoop/Compatibility
* for what changes are allowed for a *stable* .proto interface.
*/

syntax="proto2";
// This file contains protocol buffers used to communicate edits to clients
// as part of the inotify system.

Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/xattr.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/

syntax="proto2";
option java_package = "org.apache.hadoop.hdfs.protocol.proto";
option java_outer_classname = "XAttrProtos";
option java_generate_equals_and_hash = true;
Expand Down
Loading