Skip to content

Update iceberg catalog configuration for development#10439

Merged
findepi merged 1 commit intotrinodb:masterfrom
findinpath:iceberg/add-socks-proxy-setting
Jan 4, 2022
Merged

Update iceberg catalog configuration for development#10439
findepi merged 1 commit intotrinodb:masterfrom
findinpath:iceberg/add-socks-proxy-setting

Conversation

@findinpath
Copy link
Copy Markdown
Contributor

Problem description

Spin up the testing environment without docker on a Mac workstation:

 testing/bin/ptl env up --environment 'singlenode'  --config 'config-default' --without-trino

Check the schemas/tables in the iceberg connector succeeds as expected.

Try to create via trino cli a table

create table iceberg.default.temptable (id bigint);

The operation fails with the following message:

2021-12-31 11:23:14] [65536] Query failed (#20211231_102212_00000_rptjg): Failed to write manifest list file
[2021-12-31 11:23:14] org.apache.hadoop.ipc.RemoteException: File /user/hive/warehouse/temptable/metadata/snap-2694919251735337876-1-eeeba74b-42d1-4847-b1e0-f89ae40aa719.avro could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
[2021-12-31 11:23:14] 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1719)
[2021-12-31 11:23:14] 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3372)
[2021-12-31 11:23:14] 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3296)
[2021-12-31 11:23:14] 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:850)
[2021-12-31 11:23:14] 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:504)
[2021-12-31 11:23:14] 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
[2021-12-31 11:23:14] 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
[2021-12-31 11:23:14] 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
[2021-12-31 11:23:14] 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
[2021-12-31 11:23:14] 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
[2021-12-31 11:23:14] 	at java.security.AccessController.doPrivileged(Native Method)
[2021-12-31 11:23:14] 	at javax.security.auth.Subject.doAs(Subject.java:422)
[2021-12-31 11:23:14] 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
[2021-12-31 11:23:14] 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)

Implementation details

The iceberg.properties has been updated in the same fashion as the hive.properties file to contain the hive.hdfs.socks-proxy setting pointing to the port exposed by ptl-hadoop-master image.

https://github.com/trinodb/docker-images/blob/master/testing/hdp2.6-hive/Dockerfile#L42-L43

Make it usable out of the box with dockerized hadoop environment.
@findepi findepi merged commit c56b91c into trinodb:master Jan 4, 2022
@github-actions github-actions bot added this to the 368 milestone Jan 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

3 participants