HDFS FileSystem Cache Problem

[HeryLong]

Recently, I'm working on an ClosedChannelException of kafka-connect-hdfs, and I find that

1. WALFile.java have fellowing code

fs = filename.getFileSystem(conf);

...

log.error("Failed creating a WAL Reader: " + re.getMessage());
if (re.getClassName().equals(WALConstants.LEASE_EXCEPTION_CLASS_NAME)) {
  if (fs != null) {
    fs.close();
  }
}

link to WALFile.java#L146

2. StringRecordWriterProvider.java have fellowing code

final OutputStream out = path.getFileSystem(conf.getHadoopConfiguration()).create(path);

...

writer.close();

link to StringRecordWriterProvider.java#L64

So, while working under hdfs default configuration fs.hdfs.impl.disable.cache=false, will these code have effect to each other, such as one close HDFS FileSystem, which may lead to the other one get ClosedChannelException? Or, we should set fs.hdfs.impl.disable.cache=true while using this plugin always ?

Any one have some explication for this? 3ks

[cyrusv]

@HeryLong , it seems you're seeing this issue: #141 (comment)

The best solution here is to upgrade HDFS. If that's not an option, then it is recommended to turn caching off in HDFS as you suggest. H

[HeryLong]

@cyrusv
ok, got it. thank you for your comment~