Skip to content

[SPARK-29873][SQL][TESTS] Support --import directive to load queries from another test case in SQLQueryTestSuite #26497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
maropu wants to merge 4 commits into from
Closed

[SPARK-29873][SQL][TESTS] Support --import directive to load queries from another test case in SQLQueryTestSuite #26497

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8f2c066
Fix
maropu Nov 13, 2019
615347e
Fix
maropu Nov 13, 2019
93051fc
Fix
maropu Nov 13, 2019
ab0730b
Fix
maropu Nov 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/ansi/decimalArithmeticOperations.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
-- SPARK-23179: SQL ANSI 2011 states that in case of overflow during arithmetic operations,
-- an exception should be thrown instead of returning NULL.
-- This is what most of the SQL DBs do (eg. SQLServer, DB2).
--import decimalArithmeticOperations.sql
Copy link
Member

@dongjoon-hyun dongjoon-hyun Nov 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to fail still.

[info] - ansi/decimalArithmeticOperations.sql *** FAILED *** (2 seconds, 158 milliseconds)

Copy link
Member Author

@maropu maropu Nov 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, thanks for your catch-up! I've fixed now.

1 change: 1 addition & 0 deletions sql/core/src/test/resources/sql-tests/inputs/ansi/higher-order-functions.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
--import higher-order-functions.sql
9 changes: 4 additions & 5 deletions sql/core/src/test/resources/sql-tests/inputs/ansi/interval.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
-- Turns on ANSI mode
SET spark.sql.ansi.enabled=true;

select
'1' second,
2 seconds,
Expand Down Expand Up @@ -184,5 +181,7 @@ select date '2012-01-01' + interval (a + 1) day from t;

select date '2012-01-01' + (a + 1) day from t;

-- Turns off ANSI mode
SET spark.sql.ansi.enabled=false;
-- malformed interval literal with ansi mode
select 1 year to month;
select '1' year to second;
select 1 year '2-1' year to month;
2 changes: 2 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/ansi/literals.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
--- malformed interval literal with ansi mode
Copy link
Contributor

@cloud-fan cloud-fan Nov 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, do we still have interval related tests in literals.sql?

Copy link
Member Author

@maropu maropu Nov 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it ok to move all the interval-related queries here into interval.sql or ansi/interval.sql?

Copy link
Contributor

@cloud-fan cloud-fan Nov 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so. @yaooqinn do you have time to take it later? IIRC there are a few interval tests in group-by.sql as well.

Copy link
Member Author

@maropu maropu Nov 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. As for literals.sql, I'll move all the interval-related queries into correct files in this pr.

Copy link
Member

@yaooqinn yaooqinn Nov 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some cases are using interval but to verify literals such as

-- awareness of the negative sign before type
select -integer '7';
select -date '1999-01-01';
select -timestamp '1999-01-01';
select -x'2379ACFe';
select +integer '7';
select +interval '1 second';

the group-by.sql contains avg and sum support, which should be move to interval.sql

--import literals.sql
Copy link
Member

@dongjoon-hyun dongjoon-hyun Nov 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also still fails, too.

[info] - literals.sql *** FAILED *** (1 second, 989 milliseconds)

24 changes: 0 additions & 24 deletions sql/core/src/test/resources/sql-tests/inputs/decimalArithmeticOperations.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,28 +83,4 @@ select 12345678912345678912345678912.1234567 + 9999999999999999999999999999999.1
select 123456789123456789.1234567890 * 1.123456789123456789;
select 12345678912345.123456789123 / 0.000000012345678;

-- throw an exception instead of returning NULL, according to SQL ANSI 2011
set spark.sql.ansi.enabled=true;

-- test operations between decimals and constants
select id, a*10, b/10 from decimals_test order by id;

-- test operations on constants
select 10.3 * 3.0;
select 10.3000 * 3.0;
select 10.30000 * 30.0;
select 10.300000000000000000 * 3.000000000000000000;
select 10.300000000000000000 * 3.0000000000000000000;

-- arithmetic operations causing an overflow throw exception
select (5e36 + 0.1) + 5e36;
select (-4e36 - 0.1) - 7e36;
select 12345678901234567890.0 * 12345678901234567890.0;
select 1e35 / 0.1;

-- arithmetic operations causing a precision loss throw exception
select 123456789123456789.1234567890 * 1.123456789123456789;
select 123456789123456789.1234567890 * 1.123456789123456789;
select 12345678912345.123456789123 / 0.000000012345678;

drop table decimals_test;
5 changes: 0 additions & 5 deletions sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,3 @@ select transform_values(ys, (k, v) -> k + v) as v from nested;
-- use non reversed keywords: all is non reversed only if !ansi
select transform(ys, all -> all * all) as v from values (array(32, 97)) as t(ys);
select transform(ys, (all, i) -> all + i) as v from values (array(32, 97)) as t(ys);

set spark.sql.ansi.enabled=true;
select transform(ys, all -> all * all) as v from values (array(32, 97)) as t(ys);
select transform(ys, (all, i) -> all + i) as v from values (array(32, 97)) as t(ys);
set spark.sql.ansi.enabled=false;
43 changes: 43 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/interval.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,3 +88,46 @@ select justify_interval(interval '1 month -59 day 25 hour');
select justify_days(interval '1 month 59 day -25 hour');
select justify_hours(interval '1 month 59 day -25 hour');
select justify_interval(interval '1 month 59 day -25 hour');

-- interval
select interval 13.123456789 seconds, interval -13.123456789 second;
select interval 1 year 2 month 3 week 4 day 5 hour 6 minute 7 seconds 8 millisecond 9 microsecond;
select interval '30' year '25' month '-100' day '40' hour '80' minute '299.889987299' second;
select interval '0 0:0:0.1' day to second;
select interval '10-9' year to month;
select interval '20 15:40:32.99899999' day to hour;
select interval '20 15:40:32.99899999' day to minute;
select interval '20 15:40:32.99899999' day to second;
select interval '15:40:32.99899999' hour to minute;
select interval '15:40.99899999' hour to second;
select interval '15:40' hour to second;
select interval '15:40:32.99899999' hour to second;
select interval '20 40:32.99899999' minute to second;
select interval '40:32.99899999' minute to second;
select interval '40:32' minute to second;

-- ns is not supported
select interval 10 nanoseconds;

-- map + interval test
select map(1, interval 1 day, 2, interval 3 week);

-- typed interval expression
select interval 'interval 3 year 1 hour';
select interval '3 year 1 hour';

-- malformed interval literal
select interval;
select interval 1 fake_unit;
select interval 1 year to month;
select interval '1' year to second;
select interval '10-9' year to month '2-1' year to month;
select interval '10-9' year to month '12:11:10' hour to second;
select interval '1 15:11' day to minute '12:11:10' hour to second;
select interval 1 year '2-1' year to month;
select interval 1 year '12:11:10' hour to second;
select interval '10-9' year to month '1' year;
select interval '12:11:10' hour to second '1' year;

-- awareness of the positive sign before interval type
select +interval '1 second';
53 changes: 1 addition & 52 deletions sql/core/src/test/resources/sql-tests/inputs/literals.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,25 +82,6 @@ select tImEstAmp '2016-03-11 20:54:00.000';
-- invalid timestamp
select timestamp '2016-33-11 20:54:00.000';

-- interval
select interval 13.123456789 seconds, interval -13.123456789 second;
select interval 1 year 2 month 3 week 4 day 5 hour 6 minute 7 seconds 8 millisecond 9 microsecond;
select interval '30' year '25' month '-100' day '40' hour '80' minute '299.889987299' second;
select interval '0 0:0:0.1' day to second;
select interval '10-9' year to month;
select interval '20 15:40:32.99899999' day to hour;
select interval '20 15:40:32.99899999' day to minute;
select interval '20 15:40:32.99899999' day to second;
select interval '15:40:32.99899999' hour to minute;
select interval '15:40.99899999' hour to second;
select interval '15:40' hour to second;
select interval '15:40:32.99899999' hour to second;
select interval '20 40:32.99899999' minute to second;
select interval '40:32.99899999' minute to second;
select interval '40:32' minute to second;
-- ns is not supported
select interval 10 nanoseconds;

-- unsupported data type
select GEO '(10,-6)';

Expand All @@ -119,46 +100,14 @@ select X'XuZ';
-- Hive literal_double test.
SELECT 3.14, -3.14, 3.14e8, 3.14e-8, -3.14e8, -3.14e-8, 3.14e+8, 3.14E8, 3.14E-8;

-- map + interval test
select map(1, interval 1 day, 2, interval 3 week);

-- typed interval expression
select interval 'interval 3 year 1 hour';
select interval '3 year 1 hour';

-- typed integer expression
select integer '7';
select integer'7';
select integer '2147483648';

-- malformed interval literal
select interval;
select interval 1 fake_unit;
select interval 1 year to month;
select interval '1' year to second;
select interval '10-9' year to month '2-1' year to month;
select interval '10-9' year to month '12:11:10' hour to second;
select interval '1 15:11' day to minute '12:11:10' hour to second;
select interval 1 year '2-1' year to month;
select interval 1 year '12:11:10' hour to second;
select interval '10-9' year to month '1' year;
select interval '12:11:10' hour to second '1' year;
-- malformed interval literal with ansi mode
SET spark.sql.ansi.enabled=true;
select interval;
select interval 1 fake_unit;
select interval 1 year to month;
select 1 year to month;
Copy link
Contributor

@cloud-fan cloud-fan Nov 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should move some tests to ansi/interval.sql. They are not duplicated tests, they test without the leading "interval".

Copy link
Member Author

@maropu maropu Nov 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. I'll recheck later. Thanks!

select interval '1' year to second;
select '1' year to second;
select interval 1 year '2-1' year to month;
select 1 year '2-1' year to month;
SET spark.sql.ansi.enabled=false;

-- awareness of the negative sign before type
-- awareness of the negative/positive sign before type
select -integer '7';
select -date '1999-01-01';
select -timestamp '1999-01-01';
select -x'2379ACFe';
select +integer '7';
select +interval '1 second';
Loading