3.1.0

3.1.0 (2022-07-26)

Features

• add __getattr__ support to StructValue (75bded1)
• allow selection subclasses to define new node args (2a7dc41)
• api: accept Schema objects in public ibis.schema (0daac6c)
• api: add .tables accessor to BaseBackend (7ad27f0)
• api: add e function to public API (3a07e70)
• api: add ops.StructColumn operation (020bfdb)
• api: add cume_dist operation (6b6b185)
• api: add toplevel ibis.connect() (e13946b)
• api: handle literal timestamps with timezone embedded in string (1ae976b)
• api: ibis.connect() default to duckdb for parquet/csv extensions (ff2f088)
• api: make struct metadata more convenient to access (3fd9bd8)
• api: support tab completion for backends (eb75fc5)
• api: underscore convenience api (81716da)
• api: unnest (98ecb09)
• backends: allow column expressions from non-foreign tables on the right side of isin/notin (e1374a4)
• base-sql: implement trig and math functions (addb2c1)
• clickhouse: add ability to pass arbitrary kwargs to Clickhouse do_connect (583f599)
• clickhouse: implement ops.StructColumn operation (0063007)
• clickhouse: implement array collect (8b2577d)
• clickhouse: implement ArrayColumn (1301f18)
• clickhouse: implement bit aggs (f94a5d2)
• clickhouse: implement clip (12dfe50)
• clickhouse: implement covariance and correlation (a37c155)
• clickhouse: implement degrees (7946c0f)
• clickhouse: implement proper type serialization (80f4ab9)
• clickhouse: implement radians (c7b7f08)
• clickhouse: implement strftime (222f2b5)
• clickhouse: implement struct field access (fff69f3)
• clickhouse: implement trig and math functions (c56440a)
• clickhouse: support subsecond timestamp literals (e8698a6)
• compiler: restore intersect_class and difference_class overrides in base SQL backend (2c46a15)
• dask: implement trig functions (e4086bb)
• dask: implement zeroifnull (38487db)
• datafusion: implement negate (69dd64d)
• datafusion: implement trig functions (16803e1)
• duckdb: add register method to duckdb backend to load parquet and csv files (4ccc6fc)
• duckdb: enable find_in_set test (377023d)
• duckdb: enable group_concat test (4b9ad6c)
• duckdb: implement ops.StructColumn operation (211bfab)
• duckdb: implement approx_count_distinct (03c89ad)
• duckdb: implement approx_median (894ce90)
• duckdb: implement arbitrary first and last aggregation (8a500bc)
• duckdb: implement NthValue (1bf2842)
• duckdb: implement strftime (aebc252)
• duckdb: return the ir.Table instance from DuckDB's register API (0d05d41)
• mysql: implement FindInSet (e55bbbf)
• mysql: implement StringToTimestamp (169250f)
• pandas: implement bitwise aggregations (37ff328)
• pandas: implement degrees (25b4f69)
• pandas: implement radians (6816b75)
• pandas: implement trig functions (1fd52d2)
• pandas: implement zeroifnull (48e8ed1)
• postgres/duckdb: implement covariance and correlation (464d3ef)
• postgres: implement ArrayColumn (7b0a506)
• pyspark: implement approx_count_distinct (1fe1d75)
• pyspark: implement approx_median (07571a9)
• pyspark: implement covariance and correlation (ae818fb)
• pyspark: implement degrees (f478c7c)
• pyspark: implement nth_value (abb559d)
• pyspark: implement nullifzero (640234b)
• pyspark: implement radians (18843c0)
• pyspark: implement trig functions (fd7621a)
• pyspark: implement Where (32b9abb)
• pyspark: implement xor (550b35b)
• pyspark: implement zeroifnull (db13241)
• pyspark: topk support (9344591)
• sqlalchemy: add degrees and radians (8b7415f)
• sqlalchemy: add xor translation rule (2921664)
• sqlalchemy: allow non-primitive arrays (4e02918)
• sqlalchemy: implement approx_count_distinct as count distinct (4e8bcab)
• sqlalchemy: implement clip (8c02639)
• sqlalchemy: implement trig functions (34c1514)
• sqlalchemy: implement Where (7424704)
• sqlalchemy: implement zeroifnull (4735e9a)
• sqlite: implement BitAnd, BitOr and BitXor (e478479)
• sqlite: implement cotangent (01e7ce7)
• sqlite: implement degrees and radians (2cf9c5e)

Bug Fixes

• api: bring back null datatype parsing (fc131a1)
• api: compute the type from both branches of Where expressions (b8f4120)
• api: ensure that Deferred objects work in aggregations (bbb376c)
• api: ensure that nulls can be cast to any type to allow caller promotion (fab4393)
• api: make ExistSubquery and NotExistsSubquery pure boolean operations (dd70024)
• backends: make execution transactional where possible (d1ea269)
• clickhouse: cast empty result dataframe (27ae68a)
• clickhouse: handle empty IN and NOT IN expressions (2c892eb)
• clickhouse: return null instead of empty string for group_concat when values are filtered out (b826b40)
• compiler: fix bool bool comparisons (1ac9a9e)
• dask/pandas: allow limit to be None (9f91d6b)
• dask: aggregation with multi-key groupby fails on dask backend (4f8bc70)
• datafusion: handle predicates in aggregates (4725571)
• deps: update dependency datafusion to >=0.4,<0.7 (f5b244e)
• deps: update dependency duckdb to >=0.3.2,<0.5.0 (57ee818)
• deps: update dependency duckdb-engine to >=0.1.8,<0.3.0 (3e379a0)
• deps: update dependency geoalchemy2 to >=0.6.3,<0.13 (c04a533)
• deps: update dependency geopandas to >=0.6,<0.12 (b899c37)
• deps: update dependency Shapely to >=1.6,<1.8.3 (87a49ad)
• deps: update dependency toolz to >=0.11,<0.13 (258a641)
• don't mask udf module in init.py (3e567ba)
• duckdb: ensure that paths with non-extension . chars are parsed correctly (9448fd3)
• duckdb: fix struct datatype parsing (5124763)
• duckdb: force string_agg separator to be a constant (21cdf2f)
• duckdb: handle multiple dotted extensions; quote names; consolidate implementations (1494246)
• duckdb: remove timezone function invocation (33d38fc)
• geospatial: ensure that later versions of numpy are compatible with geospatial code (33f0afb)
• impala: a delimited table explicitly declare stored as textfile (04086a4), closes #4260
• impala: remove broken nth_value implementation (dbc9cc2)
• ir: don't attempt fusion when projections aren't exactly equivalent (3482ba2)
• mysql: cast mysql timestamp literals to ensure correct return type (8116e04)
• mysql: implement integer to timestamp using from_unixtime (1b43004)
• pandas/dask: look at pre_execute for has_operation reporting (cb44efc)
• pandas: execute negate on bool as not (330ab4f)
• pandas: fix struct inference from dict in the pandas backend (5886a9a)
• pandas: force backend options registration on trace.enable() calls (8818fe6)
• pandas: handle empty boolean column casting in Series conversion (f697e3e)
• pandas: handle struct columns with NA elements (9a7c510)
• pandas: handle the case of selection from a join when remapping overlapping column names (031c4c6)
• pandas: perform correct equality comparison (d62e7b9)
• postgres/duckdb: cast after milliseconds computation instead of after extraction (bdd1d65)
• pyspark: handle predicates in Aggregation (842c307)
• pyspark: prevent spark from trying to convert timezone of naive timestamps (dfb4127)
• pyspark: remove xpassing test for #2453 (c051e28)
• pyspark: specialize implementation of has_operation (5082346)
• pyspark: use empty check for collect_list in GroupConcat rule (df66acb)
• repr: allow DestructValue selections to be formatted by fmt (4b45d87)
• repr: when formatting DestructValue selections, use struct field names as column names (d01fe42)
• sqlalchemy: fix parsing and construction of nested array types (e20bcc0)
• sqlalchemy: remove unused second argument when creating temporary views (8766b40)
• sqlite: register coversion to isoformat for pandas.Timestamp (fe95dca)
• sqlite: test case with whitespace at the end of the line (7623ae9)
• sql: use isoformat for timestamp literals (70d0ba6)
• type-system: infer null datatype for empty sequence of expressions (f67d5f9)
• use bounded precision for decimal aggregations (596acfb)

Performance Improvements

• analysis: add _projection as cached_property to avoid reconstruction of projections (98510c8)
• lineage: ensure that expressions are not traversed multiple times in most cases (ff9708c)

Reverts

• ci: install sqlite3 on ubuntu (1f2705f)