[Bug]: hdfsreader读取parquet时显示INT96 is deprecated. As interim enable READ_INT96_AS_FIXED flag to read as byte array.

[jh0904]

Contact Details（联系人）

No response

What happened?

hdfsreader读取parquet时显示INT96 is deprecated. As interim enable READ_INT96_AS_FIXED flag to read as byte array.
代码如下：

{
    "job":{
        "setting":{
            "speed":{
                "channel":3,
                "bytes":-1
            }
        },
        "content":[
            {
                "reader":{
                    "name":"hdfsreader",
                    "parameter":{
                        "path":"/user/hive/warehouse/xxxx/xxxx*",
                        "defaultFS":"hdfs://127.0.0.1:xxxx",
                        "column":[
                            "*"
                        ],
                        "fileType":"parquet"
                    }
                },
                "writer":{
                    "name":"streamwriter",
                    "parameter":{
                        "print":true
                    }
                }
            }
        ]
    }
}

Version

4.0.4 (Default)

OS Type

Linux (Default)

Java JDK Version

Oracle JDK 1.8.0

Relevant log output

2021-10-28 09:53:07.204 [       job-0] ERROR Engine               - Code:[Framework-13], Description:[Addax 插件运行时出错, 具体原因请参看Addax 运行结束时的错误诊断信息 .].  - Code:[Framework-13], Description:[Addax 插件运行时出错, 具体原因请参看Addax 运行结束时的错误诊断信息 .].  - java.lang.IllegalArgumentException: INT96 is deprecated. As interim enable READ_INT96_AS_FIXED  flag to read as byte array.
	at org.apache.parquet.avro.AvroSchemaConverter$1.convertINT96(AvroSchemaConverter.java:316)
	at org.apache.parquet.avro.AvroSchemaConverter$1.convertINT96(AvroSchemaConverter.java:298)
	at org.apache.parquet.schema.PrimitiveType$PrimitiveTypeName$7.convert(PrimitiveType.java:341)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:297)
	at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:275)
	at org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:264)
	at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:134)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:185)
	at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:156)
	at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135)
	at com.wgzhao.addax.plugin.reader.hdfsreader.DFSUtil.parquetFileStartRead(DFSUtil.java:475)
	at com.wgzhao.addax.plugin.reader.hdfsreader.HdfsReader$Task.startRead(HdfsReader.java:303)
	at com.wgzhao.addax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:82)
	at java.lang.Thread.run(Thread.java:748)
 - java.lang.IllegalArgumentException: INT96 is deprecated. As interim enable READ_INT96_AS_FIXED  flag to read as byte array.
	at org.apache.parquet.avro.AvroSchemaConverter$1.convertINT96(AvroSchemaConverter.java:316)
	at org.apache.parquet.avro.AvroSchemaConverter$1.convertINT96(AvroSchemaConverter.java:298)
	at org.apache.parquet.schema.PrimitiveType$PrimitiveTypeName$7.convert(PrimitiveType.java:341)
	at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:297)
	at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:275)
	at org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:264)
	at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:134)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:185)
	at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:156)
	at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135)
	at com.wgzhao.addax.plugin.reader.hdfsreader.DFSUtil.parquetFileStartRead(DFSUtil.java:475)
	at com.wgzhao.addax.plugin.reader.hdfsreader.HdfsReader$Task.startRead(HdfsReader.java:303)
	at com.wgzhao.addax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:82)
	at java.lang.Thread.run(Thread.java:748)

[wgzhao]

感谢反馈

首先，尝试在 column 具体指定每个字段的类型，类似下面这样：

        "reader": {
          "name": "hdfsreader",
          "parameter": {
            "path": "/tmp/out_orc/*",
            "defaultFS": "hdfs://127.0.0.1:9000",
            "column": [
              {
                "index": 0,
                "type": "string"
              },
              {
                "index": 1,
                "type": "int"
              },
              {
                "index": 2,
                "type": "timestamp"
              },
              {
                "index": 3,
                "type": "boolean"
              },
              {
                "index": 4,
                "type": "string"
              }
            ],
            "fileType": "orc",
            "encoding": "UTF-8",
            "fieldDelimiter": ","
          }
        }

如果还是有上述报错，如果方便的话，请提供这个parquet给我，并提供json文件

[jh0904]

感谢回复，还是存在这个问题。相关文件在压缩包中。
json&parquet.zip
：

[jh0904]

我刚刚验证了一下，发现是由于最后的 load_time 字段 timestamp类型转换失败了，去除掉这个字段就可以正常输出。

[wgzhao]

我刚刚验证了一下，发现是由于最后的 load_time 字段 timestamp类型转换失败了，去除掉这个字段就可以正常输出。

timestamp 是新增加的支持类型，这个的确比较难搞，我在本地测试下你的情况

[wgzhao]

看的下你的文件 load_time 这个字段的时间精度是到纳秒了，这个还支持不了，也需要引入的第三方包来支持。

[wgzhao]

PARQUET-2037 解决了这个问题，我看能不能引入最新的包

[jh0904]

赞，感谢大佬耐心回复。

[wgzhao]

提供一个暂时可行的解决方案，你可以测试下：

1. 下载hdfsreader-4.0.6-SNAPSHOT.jar.zip， 解压后替换 plugin/reader/hdfsreader/hdfsreader-xxx.jar 包
2. 删除 plugin/reader/hdfsreader/hive-apache-3.1.2.jar 的软连接（该jar包自带的 parquet-avro) 版本过低
3. 将 json 文件中的第八字段的 Date 改成 timestamp
4. 测试，看结果是否符合你的要求

本地测试结果如下：

bin/addax.sh hdfs2stream.json

  ___      _     _
 / _ \    | |   | |
/ /_\ \ __| | __| | __ ___  __
|  _  |/ _` |/ _` |/ _` \ \/ /
| | | | (_| | (_| | (_| |>  <
\_| |_/\__,_|\__,_|\__,_/_/\_\

:: Addax version ::    (v4.0.6-SNAPSHOT)

2021-10-28 16:33:48.875 [        main] INFO  VMInfo               - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2021-10-28 16:33:48.894 [        main] INFO  Engine               -
{
	"content":[
		{
			"reader":{
				"parameter":{
					"path":"/tmp/out_parquet/",
					"column":[
						{
							"index":0,
							"type":"String"
						},
						{
							"index":1,
							"type":"String"
						},
						{
							"index":2,
							"type":"String"
						},
						{
							"index":3,
							"type":"String"
						},
						{
							"index":4,
							"type":"String"
						},
						{
							"index":5,
							"type":"String"
						},
						{
							"index":6,
							"type":"String"
						},
						{
							"index":7,
							"type":"timestamp"
						}
					],
					"defaultFS":"hdfs://127.0.0.1:9000",
					"fileType":"parquet"
				},
				"name":"hdfsreader"
			},
			"writer":{
				"parameter":{
					"print":true
				},
				"name":"streamwriter"
			}
		}
	],
	"setting":{
		"speed":{
			"bytes":-1,
			"channel":3
		}
	}
}

2021-10-28 16:33:48.914 [        main] INFO  PerfTrace            - PerfTrace traceId=job_-1, isEnable=false, priority=0
2021-10-28 16:33:48.914 [        main] INFO  JobContainer         - Addax jobContainer starts job.
2021-10-28 16:33:48.915 [        main] INFO  JobContainer         - Set jobId = 0
2021-10-28 16:33:48.922 [       job-0] INFO  HdfsReader$Job       - init() begin...
2021-10-28 16:33:49.045 [       job-0] INFO  DFSUtil              - hadoopConfig details:{"finalParameters":[]}
2021-10-28 16:33:49.046 [       job-0] INFO  HdfsReader$Job       - init() ok and end...
2021-10-28 16:33:49.049 [       job-0] INFO  JobContainer         - Addax Reader.Job [hdfsreader] do prepare work .
2021-10-28 16:33:49.049 [       job-0] INFO  HdfsReader$Job       - prepare(), start to getAllFiles...
2021-10-28 16:33:49.050 [       job-0] INFO  DFSUtil              - get HDFS all files in path = [/tmp/out_parquet/]
2021-10-28 16:33:50.090 [       job-0] INFO  InternalParquetRecordReader - RecordReader initialized will read a total of 418 records.
2021-10-28 16:33:50.090 [       job-0] INFO  InternalParquetRecordReader - at row 0. reading next block
2021-10-28 16:33:50.114 [       job-0] INFO  InternalParquetRecordReader - block read in memory in 24 ms. row count = 418
2021-10-28 16:33:50.370 [       job-0] INFO  DFSUtil              - [hdfs://127.0.0.1:9000/tmp/out_parquet/000000_0]是[PARQUET]类型的文件, 将该文件加入source files列表
2021-10-28 16:33:50.371 [       job-0] INFO  HdfsReader$Job       - 您即将读取的文件数为: [1], 列表为: [[hdfs://127.0.0.1:9000/tmp/out_parquet/000000_0]]
2021-10-28 16:33:50.371 [       job-0] INFO  JobContainer         - Addax Writer.Job [streamwriter] do prepare work .
2021-10-28 16:33:50.372 [       job-0] INFO  JobContainer         - Job set Channel-Number to 3 channels.
2021-10-28 16:33:50.372 [       job-0] INFO  HdfsReader$Job       - split() begin...
2021-10-28 16:33:50.372 [       job-0] INFO  JobContainer         - Addax Reader.Job [hdfsreader] splits to [1] tasks.
2021-10-28 16:33:50.372 [       job-0] INFO  JobContainer         - Addax Writer.Job [streamwriter] splits to [1] tasks.
2021-10-28 16:33:50.380 [       job-0] INFO  JobContainer         - Scheduler starts [1] taskGroups.
2021-10-28 16:33:50.386 [ taskGroup-0] INFO  TaskGroupContainer   - taskGroupId=[0] start [1] channels for [1] tasks.
2021-10-28 16:33:50.389 [ taskGroup-0] INFO  Channel              - Channel set byte_speed_limit to -1, No bps activated.
2021-10-28 16:33:50.389 [ taskGroup-0] INFO  Channel              - Channel set record_speed_limit to -1, No tps activated.
2021-10-28 16:33:50.402 [0-0-0-reader] INFO  DFSUtil              - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
2021-10-28 16:33:50.402 [0-0-0-reader] INFO  HdfsReader$Task      - read start
2021-10-28 16:33:50.403 [0-0-0-reader] INFO  HdfsReader$Task      - reading file : [hdfs://127.0.0.1:9000/tmp/out_parquet/000000_0]
2021-10-28 16:33:50.403 [0-0-0-reader] INFO  DFSUtil              - Start Read orcfile [hdfs://127.0.0.1:9000/tmp/out_parquet/000000_0].
2021-10-28 16:33:50.634 [0-0-0-reader] INFO  InternalParquetRecordReader - RecordReader initialized will read a total of 418 records.
2021-10-28 16:33:50.635 [0-0-0-reader] INFO  InternalParquetRecordReader - at row 0. reading next block
2021-10-28 16:33:50.638 [0-0-0-reader] INFO  InternalParquetRecordReader - block read in memory in 3 ms. row count = 418
2021-10-28 16:33:50.665 [0-0-0-reader] INFO  HdfsReader$Task      - end read source files...
230700	伊春市	伊春	230000	黑龙江省	黑龙江	东北	2021-10-28 01:31:56
230800	佳木斯市	佳木斯	230000	黑龙江省	黑龙江	东北	2021-10-28 01:31:56

....

820100	澳门特别行政区	澳门	820000	澳门特别行政区	澳门	华南	2021-10-28 01:31:56
2021-10-28 16:33:53.398 [       job-0] INFO  AbstractScheduler    - Scheduler accomplished all tasks.
2021-10-28 16:33:53.399 [       job-0] INFO  JobContainer         - Addax Writer.Job [streamwriter] do post work.
2021-10-28 16:33:53.400 [       job-0] INFO  JobContainer         - Addax Reader.Job [hdfsreader] do post work.
2021-10-28 16:33:53.402 [       job-0] INFO  JobContainer         - PerfTrace not enable!
2021-10-28 16:33:53.403 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 418 records, 14058 bytes | Speed 4.58KB/s, 139 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.001s |  All Task WaitReaderTime 0.253s | Percentage 100.00%
2021-10-28 16:33:53.404 [       job-0] INFO  JobContainer         -
任务启动时刻                    : 2021-10-28 16:33:48
任务结束时刻                    : 2021-10-28 16:33:53
任务总计耗时                    :                  4s
任务平均流量                    :            4.58KB/s
记录写入速度                    :            139rec/s
读出记录总数                    :                 418
读写失败总数                    :                   0

[wgzhao]

支持 INT96，要修改的地方比较多，涉及到引入的第三方的替换，赶不上这次的版本发布了。下个版本尝试解决这个问题。

[jh0904]

好的，问题已经解决了，感谢大佬支持。怒赞👍