Dataproc Templates

Dataproc templates are an effort to solve simple, but large, in-Cloud data tasks, including data import/export/backup/restore and bulk API operations. The technology under the hood which makes these operations possible is the serverless spark functionality based on Google Cloud Dataproc service.

Google is providing this collection of pre-implemented Dataproc templates as a reference and to provide easy customization for developers wanting to extend their functionality.

Templates

• HiveToBigQuery
• HiveToGCS
• PubSubToBigQuery
• GCSToBigQuery
• GCSToSpanner
• WordCount

Getting Started

Requirements

• Java 8
• Maven 3

1. Clone this repository:

    git clone https://github.com/GoogleCloudPlatform/java-docs-samples.git
   

2. Configure required properties at resources/template.properties

3. Obtain authentication credentials.

   Create local credentials by running the following command and following the oauth2 flow (read more about the command [here][auth_command]):

    gcloud auth application-default login
   

   Or manually set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to a service account key JSON file path.

   Learn more at [Setting Up Authentication for Server to Server Production Applications][ADC].

   Note: Application Default Credentials is able to implicitly find the credentials as long as the application is running on Compute Engine, Kubernetes Engine, App Engine, or Cloud Functions.

4. Format Code [Optional]

   From either the root directory or v2/ directory, run:

   mvn spotless:apply

   This will format the code and add a license header. To verify that the code is formatted correctly, run:

   mvn spotless:check

   The directory to run the commands from is based on whether the changes are under v2/ or not.

5. Building the Project

   Build the entire project using the maven compile command.

   mvn clean install

6. Executing a Template File

   Once the template is staged on Google Cloud Storage, it can then be executed using the gcloud CLI tool. The runtime parameters required by the template can be passed in the parameters field via comma-separated list of paramName=Value.

   Executing Hive to GCS template

   bin/start.sh gs://dataproc-templates/jars \
   gcp-project \
   gcp-region \
   network-subnet \
   persistent-history-server[Optional] \
   hivetogcs
   

   Executing Spanner to GCS template

   bin/start.sh gs://dataproc-templates/jars \
   gcp-project \
   gcp-region \
   network-subnet \
   persistent-history-server[Optional] \
   spannertogcs
   

Flow diagram

Below flow diagram shows execution flow for Dataproc templates:

Contributing

See the contributing instructions to get started contributing.

License

All solutions within this repository are provided under the Apache 2.0 license. Please see the LICENSE file for more detailed terms and conditions.

Disclaimer

This repository and its contents are not an official Google Product.

Contact

Questions, issues, and comments should be directed to [email protected].