Skip to content

TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null #117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
abstractdog merged 1 commit into from
Apr 10, 2021
+6 −2
Merged

TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null #117

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/TezRuntimeUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
import org.apache.hadoop.classification.InterfaceAudience.Private;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.CommonConfigurationKeys;
import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
import org.apache.tez.dag.api.TezUncheckedException;
import org.apache.tez.runtime.api.OutputContext;
import org.apache.tez.runtime.api.TaskContext;
Expand Down Expand Up @@ -269,7 +270,9 @@ public static String getBufferSizeProperty(CompressionCodec codec) {
public static String getBufferSizeProperty(String className) {
switch (className) {
case "org.apache.hadoop.io.compress.DefaultCodec":
return "io.file.buffer.size";
case "org.apache.hadoop.io.compress.BZip2Codec":
case "org.apache.hadoop.io.compress.GzipCodec":
return CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_KEY;
case "org.apache.hadoop.io.compress.SnappyCodec":
return CommonConfigurationKeys.IO_COMPRESSION_CODEC_SNAPPY_BUFFERSIZE_KEY;
case "org.apache.hadoop.io.compress.ZStandardCodec":
Expand Down
3 changes: 2 additions & 1 deletion tez-runtime-library/src/main/java/org/apache/tez/runtime/library/utils/CodecUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,8 @@ public static InputStream getDecompressedInputStreamWithBufferSize(CompressionCo
throws IOException {
String bufferSizeProp = TezRuntimeUtils.getBufferSizeProperty(codec);
Configurable configurableCodec = (Configurable) codec;
int originalSize = configurableCodec.getConf().getInt(bufferSizeProp, DEFAULT_BUFFER_SIZE);
int originalSize = bufferSizeProp == null ? DEFAULT_BUFFER_SIZE :
Copy link
Contributor

@abstractdog abstractdog Apr 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @jshmchenxi, this looks like an important bugfix
before merging this, could you please clarify what kind of codec have you used?
if bufferSizeProp is null, it means that the codec is not supported by TezRuntimeUtils.getBufferSizeProperty
if it's a codec supported by hadoop, we should add it, otherwise the DEFAULT_BUFFER_SIZE fallback makes sense

Copy link
Contributor Author

@jshmchenxi jshmchenxi Apr 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abstractdog thanks for the quick reply! I've updated the patch. In our environment the codec is GzipCodec

Copy link
Contributor

@abstractdog abstractdog Apr 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the fix, I'm about to merge this soon if no objections
@jteagles: could you please contributor rights to @jshmchenxi to the Tez project in Jira?

Copy link
Contributor

@jteagles jteagles Apr 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abstractdog, I have added contributor rights to @jshmchenxi to the Tez project in jira. I have also added you (@abstractdog) to the administrator role to the Tez which grants permissions to add contributors and other project level changes.

configurableCodec.getConf().getInt(bufferSizeProp, DEFAULT_BUFFER_SIZE);

CompressionInputStream in = null;

Expand Down