Shade hadoop compat classes so they can be loaded in hadoop jobs #13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bbeaudreault
added a commit
that referenced
this pull request
Jul 29, 2022
# This is the 1st commit message: Add maven and RPM building in blazar This uses our standard buildpack, but each module must get a custom .blazar.yaml to pass in required properties to the build. One can distribute that .blazar.yaml using build-scripts/copy_blazar_yaml.sh. In order to minimize the duplication, most of the customizations are consolidated in build-scripts/prepare_environment.sh. The RPM build is hooked into the standard build hbase-assembly, as a post-build step. The RPM build scripts in hbase-assembly/rpm-build handle turning the assembled tar.gz into multiple RPMs. We required some minor changes to the descriptor in hbase-assembly to account for the fact that our build processes each module independently instead of as a single reactor build. As a result, moduleSet does not work, so moduleSet usages were converted to dependencySet. # This is the commit message #2: Fixes to get HubSpot build to work - Our branch version munging script set-maven-versions can't parse the pom.xml due to unicode, remove it. - Our build adds build.json files to the jars. Allowlist those files in jar content verification. We could potentially try to upstream generic versions changes of these at some point # This is the commit message #3: Add client bundles for hbase2 (#11) * Add hubspot client bundles * add cdh5 mode and build the client bundles using cdh5 hadoop and java8 for now # This is the commit message #4: Add AggregationClient to client bundle (#12) # This is the commit message #5: Shade hadoop compat classes so they can be loaded in hadoop jobs (#13)
bbeaudreault
added a commit
that referenced
this pull request
Jul 29, 2022
- Add maven and RPM building in blazar - Our branch version munging script set-maven-versions can't parse the pom.xml due to unicode, remove it. - Our build adds build.json files to the jars. Allowlist those files in jar content verification. - Add client bundles for hbase2 (#11) - Add AggregationClient to client bundle (#12) - Shade hadoop compat classes so they can be loaded in hadoop jobs (#13) - Fix client bundle metrics dep issues (#14)
bbeaudreault
added a commit
that referenced
this pull request
Jul 29, 2022
- Add maven and RPM building in blazar - Our branch version munging script set-maven-versions can't parse the pom.xml due to unicode, remove it. - Our build adds build.json files to the jars. Allowlist those files in jar content verification. - Add client bundles for hbase2 (#11) - Add AggregationClient to client bundle (#12) - Shade hadoop compat classes so they can be loaded in hadoop jobs (#13) - Fix client bundle metrics dep issues (#14)
bbeaudreault
added a commit
that referenced
this pull request
Sep 6, 2022
- Add maven and RPM building in blazar - Our branch version munging script set-maven-versions can't parse the pom.xml due to unicode, remove it. - Our build adds build.json files to the jars. Allowlist those files in jar content verification. - Add client bundles for hbase2 (#11) - Add AggregationClient to client bundle (#12) - Shade hadoop compat classes so they can be loaded in hadoop jobs (#13) - Fix client bundle metrics dep issues (#14)
bbeaudreault
added a commit
that referenced
this pull request
Sep 9, 2022
- Add maven and RPM building in blazar - Our branch version munging script set-maven-versions can't parse the pom.xml due to unicode, remove it. - Our build adds build.json files to the jars. Allowlist those files in jar content verification. - Add client bundles for hbase2 (#11) - Add AggregationClient to client bundle (#12) - Shade hadoop compat classes so they can be loaded in hadoop jobs (#13) - Fix client bundle metrics dep issues (#14)
bbeaudreault
added a commit
that referenced
this pull request
Sep 9, 2022
- Add maven and RPM building in blazar - Our branch version munging script set-maven-versions can't parse the pom.xml due to unicode, remove it. - Our build adds build.json files to the jars. Allowlist those files in jar content verification. - Add client bundles for hbase2 (#11) - Add AggregationClient to client bundle (#12) - Shade hadoop compat classes so they can be loaded in hadoop jobs (#13) - Fix client bundle metrics dep issues (#14)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The comment I added to the pom.xml covers why this change is necessary and what it does. This is actually a problem that also occurred in cdh5 jobs, but the cdh5 snapshot scanning code doesn't rely on metrics at all. So trying to run with hbase2 you start to see ClassNotFoundExceptions. This change resolves those exceptions.