Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(nifi): Add dependencies for ADLS and GCP #942

Open
Jimvin opened this issue Nov 22, 2024 · 1 comment
Open

feat(nifi): Add dependencies for ADLS and GCP #942

Jimvin opened this issue Nov 22, 2024 · 1 comment

Comments

@Jimvin
Copy link
Member

Jimvin commented Nov 22, 2024 •
edited
Loading

The Stackable docker image for Apache NiFi does not contain the Hadoop Azure ADLS or Google GCS libraries. Consequently NiFi cannot write to Hive/Iceberg tables stored on ADLS/ABFS or GCS. Trying to write to an Iceberg table using the PutIceberg processor produces the following error:

PutIceberg[id=5e413c4e-7d2f-1e5a-ffff-fffffe36b8d5] Failed to load table from catalog: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found
- Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found
@Jimvin Jimvin changed the title Include Hadoop Azure libraries in Apache NiFi feat(nifi): Add dependencies for ABFS and ADLS Nov 22, 2024
Jimvin added a commit that referenced this issue Nov 22, 2024
@Jimvin
Add dependencies for ABFS and ADLS to NiFi #942
94e81f5
@Jimvin Jimvin changed the title feat(nifi): Add dependencies for ABFS and ADLS feat(nifi): Add dependencies for ADLS and GCP Nov 22, 2024
Jimvin added a commit that referenced this issue Nov 22, 2024
@Jimvin
Add dependencies for ADLS and GCP to NiFi #942
52f65ae
github-merge-queue bot pushed a commit that referenced this issue Nov 22, 2024
@Jimvin @NickLarsenNZ
Add NiFi hadoop Azure and GCP libraries (#943)
6243c6a 
* Add dependencies for ABFS and ADLS to NiFi #942

* Add dependencies for ADLS and GCP to NiFi #942

* Updated CHANGELOG.md for PR #943

* Update CHANGELOG.md

Co-authored-by: Nick <[email protected]>

* Updated CHANGELOG.md for PR #943

* chore: update changelog to move this into Unreleased

---------

Co-authored-by: Nick <[email protected]>
Co-authored-by: Nick Larsen <[email protected]>
@lfrancke
Copy link
Member

Is this fixed by #943 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lfrancke @Jimvin