Skip to content

Fix incorrect computation of reader columns for S3SelectRecordCursor#3446

Merged
martint merged 1 commit intotrinodb:masterfrom
phd3:s3-recordcursor-fix
Apr 16, 2020
Merged

Fix incorrect computation of reader columns for S3SelectRecordCursor#3446
martint merged 1 commit intotrinodb:masterfrom
phd3:s3-recordcursor-fix

Conversation

@phd3
Copy link
Copy Markdown
Member

@phd3 phd3 commented Apr 16, 2020

cc @findepi

A couple of tests in hive-tests have been failing with the following stacktrace, since #1720 was checked in.

java.lang.ArrayIndexOutOfBoundsException: 0
	at io.prestosql.plugin.hive.GenericHiveRecordCursor.isNull(GenericHiveRecordCursor.java:464)
	at io.prestosql.plugin.hive.HiveCoercionRecordCursor.isNull(HiveCoercionRecordCursor.java:155)
	at io.prestosql.plugin.hive.HiveRecordCursor.isNull(HiveRecordCursor.java:239)
	at io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:98)
	at io.prestosql.testing.MaterializedResult.materializeSourceDataStream(MaterializedResult.java:420)
	at io.prestosql.plugin.hive.AbstractTestHiveFileSystem.createTable(AbstractTestHiveFileSystem.java:454)
	at io.prestosql.plugin.hive.AbstractTestHiveFileSystem.testTableCreation(AbstractTestHiveFileSystem.java:389)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

(These tests are not executed for PRs, since tests in hive-tests require usage of AWS_ACCESS_KEY_ID defined in secrets. Right now it looks like the usage of secrets is not allowed for PRs coming from a forked repo. So tests on the PR succeed, but the commits in master are failing these tests.)

The stacktrace points to GenericHiveRecordCursor failing to get the first value in loaded variable. This indicates that GenericHiveRecordCursor was initialized with 0 columns. (Ref) This seems to happen because of a bug in S3SelectRecordCursorProvider. In case when there is no explicit projection required, an empty list of columns was being used, rather than passing along the input columns.

@cla-bot cla-bot bot added the cla-signed label Apr 16, 2020
@phd3 phd3 requested review from findepi and martint April 16, 2020 02:01
@findepi
Copy link
Copy Markdown
Member

findepi commented Apr 16, 2020

I posted #3450

@martint
Copy link
Copy Markdown
Member

martint commented Apr 16, 2020

Tests passed in #3450

@martint martint merged commit b934d9c into trinodb:master Apr 16, 2020
@martint martint added this to the 333 milestone Apr 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

3 participants