dolt 1.83.1

[BrewTestBot]

Created by brew bump

---

Created with brew bump-formula-pr.

Details

release notes

# Merged PRs
dolt

10611: Fix Parquet table import -u to skip missing table columns and match CSV warning behavior

Fix dolthub/dolt#10589

Parquet dolt table import -u no longer fails immediately when the destination table has additional columns missing from the file.
Missing destination columns now produce the standard schema mismatch warning path for Parquet.


10597: go/store: fix push latency growth for git-backed remotes

Cache remote DoltDB instances across pushes, use parented commits with bounded depth for incremental git deltas, write table files as single blobs instead of split .records/.tail intermediates, and run periodic git gc to repack cache repos.
10590: Fix nil pointer panic in workspace table Update and Delete methods

When StatementBegin encounters an error (e.g., table not found in staging root), it stores the error in wtu.err but leaves tableWriter as nil. The Update and Delete methods were dereferencing tableWriter before checking if it was nil, causing a panic.

This fix adds an early return to check for errors from StatementBegin before attempting to use tableWriter, preventing the nil pointer dereference.

go-mysql-server

3449: keep top-level plan.Limit for doltgres generate_series

PR TopN optimizations dolthub/go-mysql-server#3432 replaced plan.Limit nodes with plan.TopN, because they are unnecessary for anything in dolt and gms. However, because of the way generate_series works with LIMIT, we actually need to keep the plan.Limit at the top.

This should have minimal impact on performance
3448: Move / guard on new databases to Dolt and add sql.ErrWrongDBName and sql.SQLError to automatically add other error metadata

Parent dolthub/dolt#10431
3447: Handle Decimal types in greatest and least functions

fixes dolthub/dolt#10562
3442: sql/types: implement NumberType for system numeric types
Summary

Implement sql.NumberType for system numeric system-variable types:
SystemBoolType
systemIntType
systemUintType
systemDoubleType
Add compile-time interface conformance checks for these types.
Add tests to verify:
sql.IsNumberType recognizes system numeric types.
stats join alignment accepts system numeric types (AlignBuckets path).

Testing

go test ./sql/types ./sql/stats
go test ./sql/... -run TestDoesNotExist -tags=gms_pure_go

Related

Relates to dolthub/dolt#10535


3441: bug fix: allow ALTER TABLE DROP CONSTRAINT to remove unique indexes

When adding a unique constraint to a table, it wasn't possible to remove it using ALTER TABLE DROP CONSTRAINT. This change allows unique indexes to be removed with this syntax.

Related to: Fix ALTER TABLE ... DROP CONSTRAINT for UNIQUE constraints dolthub/doltgresql#2359
3438: Implement sql.StringType interface for systemStringType

fixes dolthub/dolt#10534

part of dolthub/dolt#10535
3436: Get index column name using offset from table name prefix length

fixes dolthub/dolt#10527

Using the index of the first found . rune as an offset to get an index column name was causing column names to not be correctly matched when the table name contained periods. Using the length of the table name as an offset avoids this issue and is also slightly more performant.
3434: /_integration/php: fix cve
3433: Correctly compute scopeLen for joins

In a join node, the scopeLen variable is the total number of columns that represent values from outer scopes.

Previously, we set this value equal to the number of columns in the immediate parent scope. When there are multiple nested scopes, this value is incorrect.

This PR adds an example of when this matters: a non-lateral join in multiple nested scopes would return an incorrect number of columns. A projection above this join would then index into this returned row, resulting in an out-of-bounds error.
3432: TopN optimizations

Changes:

Replace LIMIT(PROJECT(SORT(...))) with PROJECT(TOPN(...))
When TopN(Limit = 1), use topRowIter to avoid interface conversions and extra mallocs.

Benchmarks: TopN test dolthub/dolt#10522 (comment)


3429: Disallow creating VECTOR indexes on nullable columns

Vector indexes require NOT NULL columns, similar to spatial indexes. Previously, creating a vector index on a nullable column was silently accepted, which could cause runtime panics if any of the elements were NULL.

This adds validation in both the analyzer (CREATE TABLE) and DDL execution (CREATE INDEX / ALTER TABLE) paths, returning a clear error message.
3428: Refactor how we manage outer scopes in join iteration

There were several problems with the previous join iterator implementation:

Each type of iterator was implemented separately, even though 90% of the logic was identical. But slight variations in how they were written led to bugs that only existed in some but not others.
The merge join iterator and the full outer join iterator did not correctly handle joins within subqueries.
Some iterators would not always close child iterators.

The behavior with subqueries is the main motivation for this PR.

Previously, in order to expose values from outer scopes to iterators, we would dynamically inject PrependNodes into subquery build plans. These nodes would insert values into the beginning of returned rows, allowing parent iterators to read them and use them in expressions. To compensate for this, we would also inject StripRowNodes into joins. StripRowNodes are the opposite of PrependNodes, removing columns from their iterators.

This logic was incredibly difficult to reason about correctly:
Injecting values from the outer scope this way inherently complicates iterator logic, particularly for joins.
StripRowNodes were inserted prior to plan execution (during the assignExecIndexes pass), while PrependNodes were inserted dynamically during plan execution. Both parts of the code had to agree on how many values were being inserted/removed, which required two different packages to understand each other's inner logic. Changes to one would require changes to the other to prevent subtle bugs.
The current implementation had several bugs in the case of multiple nested scopes:

-- Lateral joins would re-include all the values from the outermost scope in the next scope, effectively doubling the number of columns with each nesting level.

-- StripRowNodes would only be generated based on the innermost scope, resulting in some injected values not being removed.

-- StripRowNodes would be generated under each join node, including between join nodes in a multi-table join. Join nodes would thus would need to re-insert these values in order to compensate... but were expected to not re-insert columns corresponding to values defined by a parent join node, except for lateral joins...  reasoning about this correctly quickly becomes untenable.

Ultimately, there's no reason why join nodes can't handle this directly. And removing the StripRowNodes type and replacing it with logic in the join iterators actually makes the logic much more consistent: Parent iterators should assume that all child iterators contain prepended values for outer scopes, and values determined by the node's schema, and nothing else. And the parent iterator returns rows that also have this property.


3427: Do not wrap hoisted subquery in Limit node until AFTER checking if it's a SubqueryAlias

Wrapping subquery nodes in a Limit node before checking if it was a SubqueryAlias was causing us to not find SubqueryAlias nodes.  This was noticed when investigating dolthub/dolt#10472. We should probably be inspecting the whole node instead of simply checking the node type, but this is a quick fix.

filed dolthub/dolt#10494 to add better test coverage and address TODOs
3426: Do not push filters into SubqueryAliases that wrap RecursiveCTEs

fixes dolthub/dolt#10472

All RecursiveCTEs are wrapped by a SubqueryAlias node. We currently don't push filters through a RecursiveCTE so pushing the filter below the SubqueryAlias just causes it to be awkwardly sandwiched in between and might make some queries less optimal if the filter contains outerscope columns since the SQA gets marked as uncacheable. We probably can push filters through a RecursiveCTE in some cases, but we should spend more time thinking about what that means (see dolthub/dolt#10490).

Also contains some refactors that I noticed along the way.

skipped tests will be fixed in #3427
3425: Do not set scope length during join planning

related to dolthub/dolt#10472

Scope length should only be set when assigning indexes if the scope length then is not zero. The scope length set during join planning is doesn't actually refer to the correct scope length, and if it was actually supposed to be zero, it was never correct set back to zero, causing a panic.
3423: Do not allow sort-based joins between text and number type columns

fixes dolthub/dolt#10435

Disables merge and range heap joins between text and number type columns
3419: Use placeholder ColumnIds for EmptyTable

Fixes dolthub/dolt#10434

EmptyTable implements TableIdNode so it was using Columns() to get the ColumnIds. EmptyTable.WithColumns() is only ever called for testing purposes; as a result, the ColSet returned is empty. This causes the column to ColumnId mapping to be incorrectly off set, leading to the wrong index id assigned.

This fix adds a case for EmptyTable in columnIdsForNode to add placeholder ColumnId values so the mappings are correctly aligned. I considered setting the actual ColSet for EmptyTable but there's actually not a good way to do that. Regardless, the index id will be set either using the name of the column or using the Projector node that wraps the EmptyTable.

Similar to SetOp, EmptyTable probably shouldn't be a TableIdNode (see dolthub/dolt#10443)
3417: Do not join remaining tables with CrossJoins during buildSingleLookupPlan

fixes dolthub/dolt#10304

Despite what the comment said, it's not safe to join remaining tables with CrossJoins during buildSingleLookupPlan. It is only safe to do so if every filter has been successfully matched to currentlyJoinedTables.  Otherwise, we end up dropping filters.

For example, we could have a query like select from A, B, inner join C on B.c0 <=> C.c0 where table A has a primary key and tables B and C are keyless. columnKey matches A's primary key column and A would be added to currentlyJoinedTables. Since the only filter references B and C and neither are part of currentlyJoinedTabes, nothing is ever added to joinCandidates. However, it's unsafe to join all the tables with CrossJoins because we still need to account for the filter on B and C.
3416: allow Doltgres to add more information schema tables
3415: Simplify Between expressions for GetField arguments

fixes dolthub/dolt#10284

part of dolthub/dolt#10340

benchmarks
3410: Replace Time.Sub call in TIMESTAMPDIFF with microsecondsDiff

fixes dolthub/dolt#10397

Time.Sub doesn't work for times with a difference greater than 9,223,372,036,854,775,807 (2^63 - 1) nanoseconds, or ~292.47 years. This is because Time.Sub returns a Duration, which is really an int64 representing nanoseconds. MySQL only stores time precision to the microsecond so we actually don't care about the difference in nanoseconds.

However, there's no easy way to directly expose the number of microseconds or seconds since epoch using the public functions for Time -- this is because seconds since epoch are encoded differently with different epochs depending on whether the time is monotonic or not (Jan 1, 1885 UTC or Jan 1, 0001 UTC).

Time.Sub uses Time.sec to normalize Time objects to seconds since the Jan 1, 0001 UTC epoch. But Time.sec isn't public so we can't call it ourselves. And Time.Second and Time.Nanosecond only give the second and nanosecond portion of a wall time, not the seconds/nanoseconds since an epoch. However, Time.UnixMicro does give us the microseconds since Unix epoch (January 1, 1970 UTC)...by calling Time.sec and then converting that to Unix time.

So microsecondsDiff calculates the difference in microseconds between two Time objects, getting their microsecond values by calling Time.UnixMicro on both of them. This isn't the most efficient but it's the best we can do with public functions.
3409: Calculate month and quarter for timestampdiff based on date and clock time values

fixes dolthub/dolt#10393

Refactors logic for year into separate function monthDiff so that it can be reused for month and quarter
3408: Added runner to hooks

This adds a sql.StatementRunner to all hooks, as they may want to execute logic depending on the hook statement. For example, cascade table deletions could literally just run DROP on the cascading items when dropping a table rather than trying to manually craft nodes which are subject to change over time.
3407: Calculate year for timestampdiff based on date and clock time values

fixes dolthub/dolt#10390

Previous calculation incorrectly assumed every year has 365 days (doesn't account for leap years) and month has 30 days (doesn't account for over half the months). The offset from this wasn't noticeable with smaller time differences but became apparent with larger time differences. This still needs to be fixed for month and quarter (#10393)
3406: Set table names to lower when creating table map

fixes dolthub/dolt#10385

3404: Bump google.golang.org/protobuf from 1.28.1 to 1.33.0

Bumps google.golang.org/protobuf from 1.28.1 to 1.33.0.



Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

[//]: # (dependabot-automerge-start)

[//]: # (dependabot-automerge-end)

Dependabot commands and options


You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show  ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/dolthub/go-mysql-server/network/alerts).


3403: /go.{mod,sum}: bump go version
3400: Allow pushing filters past Project nodes

The analyzer tries to improve query plans by moving filter expressions to be directly above the table that they reference. Previously, this had a limitation in that it would treat references to aliases in Project nodes as opaque: if the alias expression references a table, the analysis wouldn't consider the filter to reference that table.

As a result, it wasn't possible to push a filter into multiple subqueries unless a previous optimization eliminated the Project node.

This PR enhances the analysis with the following steps:

Every alias on a Project node has a unique column id, so we walk the plan tree to build a map from Project column ids to the underlying expressions.
When computing which filter expressions can be pushed to which tables, we normalize the filter expressions by replacing GetFields on a Project with the underlying expression.
When pushing a Filter above a table or into a subquery, we also replace GetFields on a Project with the underlying expression.

Reasoning about the safety is tricky here. We should replace a GetField with the underlying expression if and only if we're actually moving the Filter beneath the Project.

The main concerns would be:
If a join plan has two Project nodes without an opaque node (like a SubqueryAlias) between them, then a Filter might only get pushed beneath one Project, but references to both Projects in the filter expression could get unwrapped.
If a join plan has a Project node in one child, and a Table in the other, then a filter expression could get pushed to be above the Table even if it references the Project

In practice I don't think the first concern can happen because it would require that the filter is getting pushed to some nameable-but-not-opaque node between the Projects, which I don't think exists.

The second concern requires that the project aliases an expression that doesn't reference any tables but can't be replaced with its underlying expression in a filter, and I don't think that's possible either.


3399: GMS warning on functional indices

Makes it so create index queries with expression argument will produce a warning instead of an error.
3396: avoid converting float64 to float64

We save on conversion costs by avoiding a call to convert float64 to float64.

Unfortunately this has little to no impact on any of our benchmarks because groupbyIter uses concurrency.

benchmarks: group by sum test dolthub/dolt#10359 (comment)
3395: fewer strings.ToLower calls in gatherTableAlias

benchmarks: tolower bump dolthub/dolt#10355 (comment)
3393: add transform.InspectWithOpaque function

This changes transform.Inspect to not apply the helper function on children of sql.OpaqueNodes.

Additionally, it adds a transform.InspectWithOpaque that does.

This is to match transform.Node and transform.NodeWithOpaque.

There are still some inconsistencies between the different transform helper functions:

transform.Node:


post order (applies to node.Children then node)
no way to break out early


transform.Inspect:


pre order (applies to node then node.Children)
can break out early (only on all or none of children)
return true to continue


transform.InspectExpr:


post order (applies to expr.Children then expr)
can break out early, including stopping during children
return false to continue


3389: remove convertLeftAndRight

This PR removes c.convertLeftAndRight, which avoids calls to c.Left().Type() and c.Right().Type().

Not entirely sure why receiver methods would impact performance this much, but benchmarks say so.

Benchmarks: test revert compare dolthub/dolt#10342 (comment)
3388: Apply filter simplifications to Join condition

part of dolthub/dolt#10284

part of dolthub/dolt#10335
3387: add TODOs and update simplified Not type (wasn't expecting this branch to merge)
3386: Push filters that contain references to outer/lateral scopes.

We attempt to push filter expressions deeper into the tree so that they can reduce the size of intermediate iterators. Ideally, we want to push filters to directly above the data source that they reference.

Previously, we only pushed filters if they only referenced a single table, since pushing a filter that referenced multiple tables could potentially move the filter to a location where one of the referenced tables is no longer in scope. However, if the extra table references refer to a table in an outer scope or lateral scope, pushing the filter is completely safe. GetFields that reference an outer or lateral scope can be effectively treated as literals for the purpose of this optimization.

This PR changes getFiltersByTable, a function that maps tables onto the filters that reference those tables. Previously it would ignore filters that reference multiple tables. Now, it allows those filters provided that the extra references are to outer/lateral scopes.

This improves many of the plan tests:

The changed test in tpch_plans.go pushes a filter into the leftmost table lookup
The second changed test in query_plans.go replaces a naive InnerJoin with a CrossHashjoin
integration_plans.go shows many queries that now have an IndexedTableAccess instead of a table scan, or where we push a filter deeper into a join.

A small number of neutral / slightly negative changes:
One of the changes in integration_plans.go introduces a redundant filter that was previously being removed. In practice this is pretty benign because filters rarely impact the runtime unless they require type conversions.
The first changed test in query_plans.go replaces a LookupJoin with a LateralCrossJoin on an IndexedTableAccess. These two plans are effectively equivalent, but the LateralCrossJoin is harder to analyze, has a larger estimated cost and larger row estimate, and could in theory inhibit subsequent optimizations. I imagine we could create a new analysis pass that converts this kind of LateralCrossJoin into a LookupJoin.


3384: Look at every join node parent when computing outer scopes.

Previously, when building scopes for subqueries that appear inside joins, we would only track a single parent join node. If the subquery had multiple join parents, we would only be able to resolve references to the innermost subquery. This inhibits the optimizations we can perform.

This PR uses a custom tree walker to track a list of parent join nodes, and includes an example of a query that was not previously possible to optimize.
3383: When applying indexes from outer scopes, resolve references to table aliases

The previous implementation of the applyIndexesFromOuterScopes optimization uses the getTablesByName function to map table name references onto tables. But this function only locates ResolvedTables, and other optimizations rely on this behavior.

I've split getTablesByName into two different functions: getResolvedTablesByName, which has the original behavior, and getNamedChildren, which takes a parent node and identifies all nameable nodes in its children, including both ResolvedTables and TableAliases.
3382: Ensure join correctness for enums of different types

fixes dolthub/dolt#10311

branched from #3381

Enums of different types join based on their string value, not their underlying int value. Ensures join correctness for enums of different types by doing the following:

use type-aware conversion with enum origin types to properly convert enums to their string values
disable Lookup joins for enums of different types. Lookup joins should not work here because enums are indexed based on their int values.
disable Merge joins for enums of different types. Merge joins should not work here because enums are sorted based on their int values.


3381: Use key type for memory table index filter range

part of dolthub/dolt#10311

This change allows avoiding unnecessary type conversions when filtering by index and also simplifies some range filter expressions.

This causes foreign keys, particularly enums and sets, to be compared based on their internal values, instead of based on the generalized converted type.
3380: Generate index lookups for filters that appear in lateral joins

This PR enhances the "applyIndexesFromOuterScope" analysis pass to transform filters on tablescans into indexed table lookups, when the table's column is being compared with a column visible from a lateral join.

An example of a query that can be optimized with this change: select x, u from xy, lateral (select * from uv where y = u) uv;

Users don't often write lateral joins, but the engine can transform WHERE EXISTS expressions into lateral joins, and this lets us optimize those too.
3379: Create scope mapping for views

This is a partial fix for dolthub/dolt/issues/10297

When parsing a subquery alias, we create a new column id for each column in the SQA schema. The scope mapping is a dictionary on the SQA node that maps those column ids onto the expressions within the subquery that determine their values, and is used in some optimizations. For example, in order to push a filter into a subquery, we need to use the scope mapping to replace any GetFields that were pointing to the SQA with the expressions those fields map to. If for whatever reason the SQA doesn't have a scope mapping, we can't perform that optimization.

We parse views by recursively calling the parser on the view definition. This works but it means that the original parser doesn't have any references to the expressions inside the view, which prevents us from creating the scope mapping.

This PR attempts to fix this. Instead of defining the SQA columns in the original parser (where we no longer have access to the view's scope), we now create the columns while parsing the view, and attach them to the scope object for the view definition. Then we store that scope in a field on the Builder, so that the original parser can copy them into its own scope.

This feels hacky, but was the best way I could think of to generate the scope mappings and ensure they're visible outside the view.
3376: Add IsExpr case to replaceVariablesInExpr

Stored procedures containing IS expressions that referenced a procedure variable were breaking.
3371: Error out for NATURAL FULL JOIN

Fixes dolthub/dolt#10268

Part of dolthub/dolt#10295

Relies on dolthub/vitess#447 and adds test
3370: Wrap errgroup.Group.Go() calls for consistent panic recovery

Each spawned goroutine needs a panic recovery handler to prevent an unexpected panic from crashing the entire Go process. This change introduces a helper function that wraps goroutine creation through errgroup.Group and installs a panic recovery handler and updates existing code to use it.
3369: Added DISTINCT ON handling

Related PRs:

Added DISTINCT ON expressions dolthub/vitess#446
Added DISTINCT ON handling dolthub/doltgresql#2181


3368: Do not parse trigger bodies in LoadOnly context

fixes dolthub/dolt#10287

fixes dolthub/dolt#10288

This PR adds a LoadOnly flag to a trigger context so that the body of a CreateTrigger statement is not unnecessarily parsed. This avoids parsing the trigger body every time an insert, update, or delete is called and only parses it when the trigger is actually relevant to the event. This also avoids parsing the trigger body when show trigger is called.

This PR also rearranges some things in buildCreateTrigger to be more performant.
3367: drop schema support
3366: Fix query ok for drop view

fixes dolthub/dolt#10201
3365: Handle more types in abs

fixes dolthub/dolt#10171

fixes dolthub/dolt#10270

Add case for bool types and add default case that tries to convert value to Float64.
3364: Add panic handling to spawned goroutine

Spawning a new go routine prevented Doltgres' panic handler from catching the panic. This was discovered by Doltgres regression tests crashing from the unhandled panic.
3361: Handle empty right iterators in exists iterator as an EOF

fixes dolthub/dolt#10258

An empty right iterator in an exists iterator should be treated the same as an EOF. Previously, we were treating an empty right iterator as if it would be returning a single nil row, but this is wrong. An empty right iterator would imply an empty set, which is not the same as a single nil row.
3358: condense time.Time library calls

Instead of calling Year(), Month(), Day(), Hours(), Minutes(), and Seconds() individually, just use time.Date() and time.Clock().

Benchmarks: faster date parse bump dolthub/dolt#10249 (comment)
3356: Return boolean literal when simplifying AND and OR filters

fixes dolthub/dolt#10243

Previously, if one of the children of an AND or OR expression was a literal that evaluated to false or true (respectively), we would return the child. However, this caused a typing issue since AND and OR are booleans while its children can be of any type. Take the expression 19 or 's' for example; the left child 19 evaluates to true, making the expression true every time, but returning the left child 19 as a simplification would be incorrect since we would want the expression to simplify to a boolean true, not the number 19.

Now, instead of returning the child that determines the result of an AND or OR expression, we return the boolean value that the express...

View the full release notes at https://github.com/dolthub/dolt/releases/tag/v1.83.1.

---

[github-actions]

🤖 An automated task has requested bottles to be published to this PR.

Caution

Please do not push to this PR branch before the bottle commits have been pushed, as this results in a state that is difficult to recover from. If you need to resolve a merge conflict, please use a merge commit. Do not force-push to this PR branch.