feat: Introduce sqllogictest

[Weijun-H]

Ticket(s) Closed

• Closes Integrate sqllogictest #144

What

Why

How

• introduce sqllogictest
• convert some existing tests to .slt
• Add CI to trigger sqllogictest
• Add README about how to use sqllogictest

Tests

[philippemnoel]

We an probably remove this, we don't plan to have different versions

[philippemnoel]

Could you please explain what sqllogictest does better than our existing test suite? Is it a replacement, or an addition to our current test suite?

This all looks quite complicated and a lot more code to maintain. What is missing from our current test suite that justifies this?

[Weijun-H]

Could you please explain what sqllogictest does better than our existing test suite? Is it a replacement, or an addition to our current test suite?

This all looks quite complicated and a lot more code to maintain. What is missing from our current test suite that justifies this?

I think sqllogictest is a better addition to the current test suite. Because

1. Ease of maintenance: As projects grow, test suites often become more complex and harder to maintain. Sqllogictest's standardized format can simplify the process of adding new tests and updating existing ones, potentially reducing the maintenance burden.
2. Scalability: For larger projects with numerous complex SQL queries, sqllogictest's approach may be more manageable than traditional unit tests. Its design is geared towards handling a large volume of SQL-specific tests efficiently.
3. Industry adoption: The fact that other database projects like DuckDB, Datafusion, Databend, and RisingWave use sqllogictest is a strong endorsement of its value for SQL-heavy applications. This widespread adoption suggests that it's a proven solution for SQL testing at scale.
4. Complementary approach: Using sqllogictest alongside your current rstest suite can provide a more comprehensive testing strategy. Rstest can continue to handle unit tests and specific Rust-related scenarios, while sqllogictest focuses on SQL integration testing.
5. Gradual adoption: You can start by implementing sqllogictest for new features or particularly complex areas of your codebase, gradually expanding its use as you see benefits, without immediately replacing your entire existing test suite.

Additionally, after #107 is completed, our sqllogictest test suite could use COPY TO to test by customized foreign tables.

For example

statement ok
COPY (SELECT 42 AS a, 'hello' AS b) TO 'query.parquet' (FORMAT PARQUET);

statement ok
CREATE FOREIGN DATA WRAPPER parquet_wrapper
HANDLER parquet_fdw_handler VALIDATOR parquet_fdw_validator;

statement ok
CREATE SERVER parquet_server FOREIGN DATA WRAPPER parquet_wrapper;

statement ok
CREATE FOREIGN TABLE trips ()
SERVER parquet_server
OPTIONS (files 'test_files/scratch/query.parquet');

# specific testing
statement ok
CREATE VIEW query_view AS
SELECT * FROM query_table;

......

[philippemnoel]

Could you please explain what sqllogictest does better than our existing test suite? Is it a replacement, or an addition to our current test suite?
This all looks quite complicated and a lot more code to maintain. What is missing from our current test suite that justifies this?

I think sqllogictest is a better addition to the current test suite. Because

1. Ease of maintenance: As projects grow, test suites often become more complex and harder to maintain. Sqllogictest's standardized format can simplify the process of adding new tests and updating existing ones, potentially reducing the maintenance burden.
2. Scalability: For larger projects with numerous complex SQL queries, sqllogictest's approach may be more manageable than traditional unit tests. Its design is geared towards handling a large volume of SQL-specific tests efficiently.
3. Industry adoption: The fact that other database projects like DuckDB, Datafusion, Databend, and RisingWave use sqllogictest is a strong endorsement of its value for SQL-heavy applications. This widespread adoption suggests that it's a proven solution for SQL testing at scale.
4. Complementary approach: Using sqllogictest alongside your current rstest suite can provide a more comprehensive testing strategy. Rstest can continue to handle unit tests and specific Rust-related scenarios, while sqllogictest focuses on SQL integration testing.
5. Gradual adoption: You can start by implementing sqllogictest for new features or particularly complex areas of your codebase, gradually expanding its use as you see benefits, without immediately replacing your entire existing test suite.

Additionally, after #107 is completed, our sqllogictest test suite could use COPY TO to test by customized foreign tables.

For example

statement ok
COPY (SELECT 42 AS a, 'hello' AS b) TO 'query.parquet' (FORMAT PARQUET);

statement ok
CREATE FOREIGN DATA WRAPPER parquet_wrapper
HANDLER parquet_fdw_handler VALIDATOR parquet_fdw_validator;

statement ok
CREATE SERVER parquet_server FOREIGN DATA WRAPPER parquet_wrapper;

statement ok
CREATE FOREIGN TABLE trips ()
SERVER parquet_server
OPTIONS (files 'test_files/scratch/query.parquet');

# specific testing
statement ok
CREATE VIEW query_view AS
SELECT * FROM query_table;

......

Got it. That's convincing. Alright, then! I'm in favour of it, especially after splitting our tests into their own folder/crate. Thank you for doing this and for pushing for it!

[philippemnoel]

At this time, we've decided this is not a priority.