Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- **`sqlp`: new SQL surface from the Polars bump to `9d5804d`** (upstream `py-1.44.0` -> the pinned rev). None of this needed qsv changes beyond the bump, but none of it was covered by tests either, so `tests/test_sqlp.rs` now exercises each one:
- **`GROUP BY GROUPING SETS (...)` / `ROLLUP(...)` / `CUBE(...)`**, plus **`GROUPING(k, ...)`** and **`GROUPING_ID(k, ...)`** to tell a subtotal row's NULL marker apart from a NULL in the data ([#29278](https://github.com/pola-rs/polars/pull/29278)). Both pack one bit per argument, MSB first, so argument order matters. Pair them with `--wnull-value` - a subtotal's NULL keys are otherwise written as empty fields.
- **the bare date-part functions `YEAR`, `QUARTER`, `MONTH`, `WEEK`, `DAY`/`DAYOFMONTH`, `DAYOFWEEK`, `DAYOFYEAR`, `HOUR`, `MINUTE`, `SECOND`** ([#29269](https://github.com/pola-rs/polars/pull/29269)). `WEEK` and `DAYOFWEEK` are ISO-based, which is the part worth knowing: `WEEK('2024-12-31')` is **1** (ISO week 1 of 2025), not 53, and `DAYOFWEEK` counts Monday as 1.
- **`DATE '...'` / `TIMESTAMP '...'` typed literals** ([#29007](https://github.com/pola-rs/polars/pull/29007)), and **date +/- integer arithmetic** shifting by whole days with the integer on either side, from a literal or a column, plus **Decimal comparisons in non-equi joins** ([#29156](https://github.com/pola-rs/polars/pull/29156)).
- **parenthesized `JOIN ... ON (...)` constraints** ([#28967](https://github.com/pola-rs/polars/pull/28967)); relation aliases declared *inside* those parens now resolve in the outer SELECT, since bare parens around a join do not open a new scope ([#29158](https://github.com/pola-rs/polars/pull/29158)).
- **`OVER` on multi-argument aggregates** - `CORR`, `COVAR_POP`, `COVAR_SAMP`, `QUANTILE_CONT`, `QUANTILE_DISC`, `STRING_AGG` ([#29160](https://github.com/pola-rs/polars/pull/29160)) - and **`NULLS FIRST`/`NULLS LAST` inside a window's own `ORDER BY`** ([#29159](https://github.com/pola-rs/polars/pull/29159)) - though only when the window has a **single** sort key: with two or more keys the `NULLS` clause is silently ignored and the nulls follow the sort direction instead, which we reported upstream as [#29390](https://github.com/pola-rs/polars/issues/29390).
- **`EXISTS`/`NOT EXISTS` and correlated scalar subqueries**, CTEs that shadow a registered table, case-insensitive relation names, and `ORDER BY` over an aggregate that is not in the SELECT list ([#29006](https://github.com/pola-rs/polars/pull/29006), [#29010](https://github.com/pola-rs/polars/pull/29010)).
- `sqlp`: **`APPROX_QUANTILE(col, q [, allowed_rank_error [, method]])`** - an approximate quantile aggregate, new to the Polars SQL frontend in [pola-rs/polars#29288](https://github.com/pola-rs/polars/pull/29288). Upstream cfg-gates it behind a `approx_quantile` feature that ships only inside their `docs-selection`/`full` umbrellas, so qsv now enables it explicitly - without that, `sqlp` rejected the function as `unsupported function 'approx_quantile'`. `allowed_rank_error` defaults to `0.001` and `method` selects the sketch (e.g. `'kll'`); anything outside 2-4 arguments is a syntax error rather than a silent fallback. Being sketch-based it trades exactness for speed on large inputs - use `QUANTILE_CONT`/`QUANTILE_DISC` when you need the exact value.
- **gallery: a real per-capita county rate map where the rate does NOT invert the ranking.** The gallery's two existing `--denominator census` figures both show a raw-count choropleth whose ranking *inverts* once divided by population - `district_requests` on six hand-drawn rectangles, `tristate_county_incidents` on 210 real counties with counts constructed to be anti-correlated with population. Both teach the lesson well and both are synthetic in exactly the place that matters, so a reader could fairly conclude that per-capita always flips a map. `wpa_211_requests` is the counter-example, on data that is real all the way down: **279,464 2-1-1 helpline requests** across the **25 Western Pennsylvania counties** served by the United Way of Southwestern Pennsylvania - the upstream resource *entire*, 2023-09-22 to 2025-08-27, no window and no sample (via the [WPRDC](https://data.wprdc.org/dataset/211-requests)). The count panel is Allegheny County and almost nothing else - **172,035 requests, 61.6% of the file** - and the rate panel beside it leaves Allegheny on top at **138.9 per 1,000 residents**. What moves is the middle: **Venango climbs 11th to 3rd**, **Cambria 5th to 2nd**, **Butler falls 8th to 13th**, and the two top-eights share only 6 of 8 members. Per-capita is a *different question*, not a trick that flips a map. The figure also documents the caveat the other two cannot: a rate map of a *helpline* measures service reach as much as need, and the map says so itself: its darkest county is **McKean, at 2 requests against 39,904 residents** - 0.05 per 1,000, where the next-lowest county (Elk) is 12.3, some 245 times higher. Nobody believes McKean has no hardship; it sits at the edge of *this* call center's intake, so the figure is measuring who dials 2-1-1, not who needs it. A per-capita map inherits whatever its numerator was actually counting. Nothing is stored in the CSV but the requests themselves: `--geojson auto` fetches the county boundaries **and** canonicalizes the county *names* to Census GEOIDs, and `--denominator census@2024` fetches ACS 5-year total population (`B01003`), so there is no FIPS column and no committed boundary file. The upstream feed's `zip_code` is dropped rather than mis-tagged (a second geo concept would add a competing choropleth candidate), and the dataset's older 2020-2023 resource is deliberately not concatenated: 5.4% of its rows carry a pipe-delimited `Allegheny County|Westmoreland County` multi-county value, and every way of resolving those changes the map. Built by the new `examples/viz/gen_wpa_211_requests.py`, which asserts its own row count, county count and date span so a silently revised upstream feed fails instead of quietly re-cutting the figure the caption describes.
- `fetch` & `fetchpost`: **`--default-encoding <enc>`** - the fallback character encoding used to decode a response body when the server sends **no** `charset` parameter in its `Content-Type` header. Takes [WHATWG encoding labels](https://encoding.spec.whatwg.org/#names-and-labels) (`utf-8`, `windows-1252`, `iso-8859-1`, `shift_jis`, …) and defaults to `utf-8`, so behavior is unchanged unless you ask for it. A `charset` the server *does* send always wins; this only fills the gap for legacy APIs that serve latin-1/windows-1252 with a bare `Content-Type`. Unknown labels are rejected up front rather than silently falling back to UTF-8. The value participates in the cross-session disk/Redis cache key, so re-running the same URL under a different encoding re-decodes instead of serving the previous run's text. **Adding the encoding to that key invalidates existing `fetch`/`fetchpost` disk and Redis cache entries**, including for users who never pass the flag - a warm cache is re-fetched once after upgrading, and the orphaned entries age out on their normal TTL.

### Changed
- **`sqlp`: three Polars SQL behavior changes that can break an existing query.** All three arrive with the bump to `9d5804d` and are now pinned by tests:
- **`CAST(<string> AS DATE/TIME/TIMESTAMP)` parses rather than casts.** The string->temporal cast kernel was removed upstream ([#28062](https://github.com/pola-rs/polars/pull/28062)) and SQL casts were re-routed through format inference ([#28986](https://github.com/pola-rs/polars/pull/28986)), so the spelling still works - but `CAST` is now **strict** and fails the whole query on a value it cannot parse, where `TRY_CAST` nulls just that row. The failure has two shapes: if *some* rows parse you get ``conversion from `str` to `date` failed``, naming the offending value; if *none* parse you get `could not find an appropriate format to parse dates`.
- **SQL scope rules follow Postgres** ([#28937](https://github.com/pola-rs/polars/pull/28937)). Once a relation is aliased, its original name is out of scope: `SELECT t1.a FROM t1 AS f` now errors with `no table or struct column named 't1' found`. An unqualified `ORDER BY` key still resolves, and an outer alias is still visible to a correlated subquery.
- **a scalar subquery's aggregate binds to its own relation** ([#28939](https://github.com/pola-rs/polars/pull/28939)), so aggregating an outer-relation column inside the subquery is an error instead of silently resolving; and **unaliased constants in a `SELECT` with `GROUP BY`** are projected once per group as `literal`, `literal:1`, `literal:2`, ... rather than colliding on one column ([#29367](https://github.com/pola-rs/polars/pull/29367)).
- **`reqwest`'s `charset` and `system-proxy` features are now enabled explicitly.** Both were already active for the full `qsv` binary, but only by accident of Cargo feature unification: `cpc` (via `apply`), `plotly_static` (via `viz_static`) and `polars-io` all depend on `reqwest` *without* `default-features = false`, and reqwest's defaults include them. Listing them on qsv's own dependency pins that behavior, and turns it on for the leaner binaries that nothing else was pulling it into - **`qsvlite`, `qsvdp` and `qsvmcp` now**:
- decode HTTP response bodies according to the `charset` parameter of the response's `Content-Type` (with BOM sniffing and stripping) instead of a blind lossy-UTF-8 conversion, and
- honor **OS-level proxy configuration** (macOS SystemConfiguration, Windows registry) in addition to the `HTTP_PROXY`/`HTTPS_PROXY` environment variables.
Expand Down
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,11 @@ rust-i18n = { version = "4.2.1", optional = true }
pragmastat = "14.0"
polars-readstat-rs = { version = "0.23", optional = true }
polars = { version = "0.55", features = [
# "approx_quantile" powers the APPROX_QUANTILE SQL aggregate in sqlp. It is
# NOT in any umbrella qsv enables (upstream keeps it in docs-selection/full
# only), so without it polars-sql cfg-gates the function out entirely and
# sqlp rejects it as "unsupported function 'approx_quantile'".
"approx_quantile",
"asof_join",
"avro",
# "avx512" is x86_64-only — enabled via the
Expand Down
Loading
Loading