Releases: dolthub/dolt
Releases · dolthub/dolt
1.43.16
Merged PRs
dolt
- 8576: Change LateBindQueryist to use
mysql.NewSalt()
In preparation for refactoring the auth server interface in Vitess/GMS, this change makes LateBindQueryist callmysql.NewSalt()
directly, instead of relying on theNewSalt()
method on theMySQLDb
instance (which internally callsmysql.NewSalt()
).
go-mysql-server
- 2753: Fixes CTE issue with auth
This fixes the test introduced in:- dolthub/go-mysql-server#2745
Related PR: - dolthub/vitess#377
- dolthub/go-mysql-server#2745
- 2750: Fix alter NULL enum value panic
- 2739: Embed the
embedded.Tracer
into memTracer, to fulfil the updated interface
I was trying to use this projectgo-mysql-server
as a dependency alongside github.com/grafana/grafana-plugin-sdk-go/data. But the latter project depends on:- go.opentelemetry.io/otel v1.31.0
- go.opentelemetry.io/otel/trace v1.31.0
When I updated my go modules withgo mod tidy
I started to hit a compilation error:
Hence this change 🙏# github.com/dolthub/go-mysql-server/test /Users/samjewell/go/pkg/mod/github.com/dolthub/[email protected]/test/mem_tracer.go:22:2: could not import go.opentelemetry.io/otel/trace/embedded (open : no such file or directory) FAIL github.com/grafana/poc-sql-expressions-embedded-sql/dolthub-go-mysql-server [build failed] FAIL
vitess
- 378: support
set names binary
syntax (no quotes)
syntax for: #8574 - 377: Fixed CTE issue with auth
Fixes the test introduced in: - 375: support multiple like tables
Closed Issues
- 8574:
set names binary
fails.set names 'binary'
succeeds. Seems unquoted collations in MySQL are supposed to work withset names
Performance
Read Tests | MySQL | Dolt | Multiple |
---|---|---|---|
covering_index_scan | 2.14 | 0.62 | 0.3 |
groupby_scan | 13.22 | 17.01 | 1.3 |
index_join | 1.34 | 2.3 | 1.7 |
index_join_scan | 1.27 | 1.89 | 1.5 |
index_scan | 34.33 | 55.82 | 1.6 |
oltp_point_select | 0.18 | 0.27 | 1.5 |
oltp_read_only | 3.49 | 5.57 | 1.6 |
select_random_points | 0.34 | 0.67 | 2.0 |
select_random_ranges | 0.39 | 0.67 | 1.7 |
table_scan | 34.95 | 55.82 | 1.6 |
types_table_scan | 75.82 | 144.97 | 1.9 |
reads_mean_multiplier | 1.5 |
Write Tests | MySQL | Dolt | Multiple |
---|---|---|---|
oltp_delete_insert | 7.98 | 5.88 | 0.7 |
oltp_insert | 3.75 | 2.91 | 0.8 |
oltp_read_write | 8.58 | 11.45 | 1.3 |
oltp_update_index | 3.82 | 2.97 | 0.8 |
oltp_update_non_index | 3.89 | 2.91 | 0.7 |
oltp_write_only | 5.28 | 5.88 | 1.1 |
types_delete_insert | 7.7 | 6.21 | 0.8 |
writes_mean_multiplier | 0.9 |
TPC-C TPS Tests | MySQL | Dolt | Multiple |
---|---|---|---|
tpcc-scale-factor-1 | 98.98 | 40.93 | 2.4 |
tpcc_tps_multiplier | 2.4 |
Overall Mean Multiple | 1.60 |
---|
1.43.15
Merged PRs
dolt
- 8567: patch with tag collision test
Tag collisions exists and can lead to false positives. - 8566: Bug fix:
dolt_commit_diff
support for detached head mode
When we added support for showing diffs for theSTAGED
root throughdolt_commit_diff_
system tables, we didn't take into account that in detached head mode, there wouldn't be a valid staged root. This change detects if we're in detached head mode and replaces the staged root with the root of the detached head. - 8541: reverse keyless indexscans apply reversal
re: dolthub/go-mysql-server#2737
go-mysql-server
- 2751: Fix drop view if exists for doltgres
- 2750: Fix alter NULL enum value panic
- 2749: Support create and drop view with schema name
- 2748: use
equal
method for scopecolumn
This fixesDoltgresType
comparison issue - 2747: Add schema name to definitions for views, triggers, procedures
- 2736: use Equals method for sql.Types
The newDoltgresType
struct cannot be compared with==
, so should usesql.Type.Equals()
function.
Closed Issues
Performance
Read Tests | MySQL | Dolt | Multiple |
---|---|---|---|
covering_index_scan | 2.07 | 0.62 | 0.3 |
groupby_scan | 13.22 | 16.71 | 1.3 |
index_join | 1.37 | 2.3 | 1.7 |
index_join_scan | 1.27 | 1.86 | 1.5 |
index_scan | 34.33 | 55.82 | 1.6 |
oltp_point_select | 0.18 | 0.27 | 1.5 |
oltp_read_only | 3.49 | 5.37 | 1.5 |
select_random_points | 0.34 | 0.65 | 1.9 |
select_random_ranges | 0.39 | 0.64 | 1.6 |
table_scan | 34.95 | 55.82 | 1.6 |
types_table_scan | 75.82 | 144.97 | 1.9 |
reads_mean_multiplier | 1.5 |
Write Tests | MySQL | Dolt | Multiple |
---|---|---|---|
oltp_delete_insert | 8.13 | 5.88 | 0.7 |
oltp_insert | 3.82 | 2.91 | 0.8 |
oltp_read_write | 8.58 | 11.45 | 1.3 |
oltp_update_index | 3.89 | 2.97 | 0.8 |
oltp_update_non_index | 3.89 | 2.91 | 0.7 |
oltp_write_only | 5.47 | 5.88 | 1.1 |
types_delete_insert | 7.84 | 6.21 | 0.8 |
writes_mean_multiplier | 0.9 |
TPC-C TPS Tests | MySQL | Dolt | Multiple |
---|---|---|---|
tpcc-scale-factor-1 | 99.82 | 41.45 | 2.4 |
tpcc_tps_multiplier | 2.4 |
Overall Mean Multiple | 1.60 |
---|
1.43.14
Merged PRs
dolt
- 8555: In order to support multiple index types, create interface types for prolly Maps, and make the tree.MutableMap class generic.
- 8554: use new decimal type for
dolt diff
When modifying column from a decimal with a certain scale/precision to another decimal with different scale/precision, we should display diffs using the new decimal type instead of the old one.
related: #8359
fixes: #8551 - 8542: add GetDatabaseName function to DoltDB
- 8540: add a databasename parameter to the DoltDBFromCS
- 8538: Fix JSON merge issue that would report imprecise diffs in some situations
If:- A chunk begins with an object "A"
- A value "A.b" within this object was modified
- The previous chunk was also modified
Then the differ would incorrectly report that the entire "A" object had been modified, instead of the sub-value "A.b"
This could then lead to situations where the merge cannot be auto-resolved, because the differ claims that an object has been modified divergently, when it's actually two unrelated fields in the object that have been modified.
This PR fixes that scenario by properly detecting when the next chunk marks the start of an object.
- 8527: Support multiple schema for system tables with $tablename for doltgres
go-mysql-server
- 2741: fix: store the hashed password to 'authentication_string' (to #2740)
This PR addresse the issue #2740.- resolve #2740
- 2738: support multi-inherit
syntax: dolthub/vitess#375 - 2736: use Equals method for sql.Types
The newDoltgresType
struct cannot be compared with==
, so should usesql.Type.Equals()
function. - 2734: always unwrap mysql priv db
fixes: #8530 - 2733: disabling read-comitted
- 2721: Additional support for vector indexes.
This PR adds the vector function VEC_DISTANCE to GMS, as well as support for adding and altering vector indexes, and support forSHOW CREATE
on tables with vector indexes.
Vector indexes are not yet supported in Dolt. The corresponding version bump in Dolt will have checks preventing vector indexes from being added to Dolt tables.
vitess
Closed Issues
1.43.13
Merged PRs
dolt
- 8534: Fix issue where JSON diff would fail under specific circumstances.
If all of the following are true during a JSON diff operation:- One document fits in a single chunk and the other doesn't
- The location of the chunk boundary in the larger document is also present in the smaller document. (It doesn't correspond to a location that was added or removed)
- The chunk boundary falls at the end of a value in the document (before the next key or comma/right brace/etc)
Then the differ would fail to advance the prolly tree cursor and would incorrectly see the larger document as corrupt.
This fixes that issue and improves the error messaging to make it seem less like the database is corrupt, since the error is much more likely to be a parsing bug like this one.
Closed Issues
- 8530: Materialized View Gives PrivilegedDatabase Error
Performance
Read Tests | MySQL | Dolt | Multiple |
---|---|---|---|
covering_index_scan | 2.11 | 0.61 | 0.3 |
groupby_scan | 12.98 | 16.71 | 1.3 |
index_join | 1.37 | 2.26 | 1.6 |
index_join_scan | 1.3 | 1.82 | 1.4 |
index_scan | 34.33 | 55.82 | 1.6 |
oltp_point_select | 0.18 | 0.27 | 1.5 |
oltp_read_only | 3.49 | 5.37 | 1.5 |
select_random_points | 0.34 | 0.65 | 1.9 |
select_random_ranges | 0.39 | 0.64 | 1.6 |
table_scan | 34.95 | 55.82 | 1.6 |
types_table_scan | 75.82 | 147.61 | 1.9 |
reads_mean_multiplier | 1.5 |
Write Tests | MySQL | Dolt | Multiple |
---|---|---|---|
oltp_delete_insert | 7.98 | 5.77 | 0.7 |
oltp_insert | 3.75 | 2.91 | 0.8 |
oltp_read_write | 8.58 | 11.24 | 1.3 |
oltp_update_index | 3.89 | 2.91 | 0.7 |
oltp_update_non_index | 3.89 | 2.86 | 0.7 |
oltp_write_only | 5.37 | 5.88 | 1.1 |
types_delete_insert | 7.7 | 6.09 | 0.8 |
writes_mean_multiplier | 0.9 |
TPC-C TPS Tests | MySQL | Dolt | Multiple |
---|---|---|---|
tpcc-scale-factor-1 | 99.97 | 41.75 | 2.4 |
tpcc_tps_multiplier | 2.4 |
Overall Mean Multiple | 1.60 |
---|
1.43.12
Merged PRs
dolt
- 8526: Avoid comparing
sql.Types
indolt diff
Fixes #8511
Not all implementations ofsql.Type
are comparable. In particular, the enum type has a map field which makes the entire type incomparable.
To avoid doing this comparison in thechooseMostFlexibleType
function, we modify it to compare the underlying type tag instead. This PR also short-circuits computing a unified schema unless the schema has changed. - 8514: Better view caching
Binding a table name to a catalog symbol previously always loaded the schemas table from disk to first attempt binding a view. Now we cachedolt_schemas
using its hash. If the table does not exist, no views are defined, and we short circuit attempting to bind a name to a view.
go-mysql-server
- 2733: disabling read-comitted
- 2732: Get field string concat
Replacefmt.Sprintf
with string concat (+).BenchmarkSprintf-12 18748582 63.29 ns/op BenchmarkAddString-12 39934550 27.61 ns/op
- 2729: move
applyEventScheduler
logic andeventscheduler
to builder - 2712: Planbuilder Authorization
- dolthub/vitess#372
What's missing: - SHOW commands aren't in yet
information_schema
doesn't restrict it's output yet- Need far more robust testing than what currently exists
I thinkSHOW
andinformation_schema
will probably have the same solution, which may be to continue doing what we were doing before. Besides that, pretty much every works according to our current tests (outside of the aforementioned missing items).
- dolthub/vitess#372
vitess
- 374: USE Revisions use proper auth
- 373: Fixed printing bug in AliasedExpr node (was calling wrong print metho…
…d, which doesn't work with custom formatters) - 372: Embedded Authorization
Closed Issues
- 8511: Panic on
dolt show
1.43.11
Merged PRs
dolt
- 8522: libraries/doltcore/remotestorage: Improve connection reuse when fetching chunks from remote storage.
Improves performance of fetches from DoltHub, sql-server, and doltlab.
Improves some situations where routers and network access points do not respond well to clients which open a great number of connections to the same TCP endpoint, many of which live for a very short period of time.
http.DefaultClient comes with a default MaxIdleConnsPerHost of 2. Fetching with a high number of concurrent downloads results in many connections not being reused efficiently because they cannot get back into the idle pool because it is so small. Here we increase the MaxIdleConnsPerHost to be large enough so that our active connections during a fetch will fit within it.
Note: S3 limits a single HTTP connection to 100 requests, after which it is closed server-side. So currently dolt can still require a large number of new connections over the course of a pull, and the rate at which they are opened will depend on a number of factors including available throughput, round trip time resolving storage locations, etc. But this change will always be a big improvement over the old behavior.
go-mysql-server
- 2729: move
applyEventScheduler
logic andeventscheduler
to builder - 2727: fix dropped error in createView
- 2719: move
applyUpdateAccumulators
This PR deletes theplan.RowUpdateAccumulator
node, and directly injects anaccumulatorIter
into the rowexec tree infinalizeIters
.
Doltgres fix: dolthub/doltgresql#909
Closed Issues
Performance
Read Tests | MySQL | Dolt | Multiple |
---|---|---|---|
covering_index_scan | 2.07 | 0.62 | 0.3 |
groupby_scan | 13.7 | 16.71 | 1.2 |
index_join | 1.39 | 2.35 | 1.7 |
index_join_scan | 1.27 | 1.89 | 1.5 |
index_scan | 34.33 | 54.83 | 1.6 |
oltp_point_select | 0.18 | 0.27 | 1.5 |
oltp_read_only | 3.49 | 5.47 | 1.6 |
select_random_points | 0.34 | 0.67 | 2.0 |
select_random_ranges | 0.39 | 0.64 | 1.6 |
table_scan | 34.33 | 54.83 | 1.6 |
types_table_scan | 75.82 | 144.97 | 1.9 |
reads_mean_multiplier | 1.5 |
Write Tests | MySQL | Dolt | Multiple |
---|---|---|---|
oltp_delete_insert | 8.13 | 5.88 | 0.7 |
oltp_insert | 3.82 | 2.91 | 0.8 |
oltp_read_write | 8.58 | 11.45 | 1.3 |
oltp_update_index | 3.82 | 2.97 | 0.8 |
oltp_update_non_index | 3.89 | 2.91 | 0.7 |
oltp_write_only | 5.37 | 5.99 | 1.1 |
types_delete_insert | 7.7 | 6.32 | 0.8 |
writes_mean_multiplier | 0.9 |
TPC-C TPS Tests | MySQL | Dolt | Multiple |
---|---|---|---|
tpcc-scale-factor-1 | 99.4 | 40.78 | 2.4 |
tpcc_tps_multiplier | 2.4 |
Overall Mean Multiple | 1.60 |
---|
1.43.10
Merged PRs
dolt
- 8508: Update more system tables for doltgres
- 8505: Fix panics on schema changes
fixes: #8504
Use schema hash to detect when database statistics might be incompatible. Drops all statistics for schema update. This could be improved to a subset of statistics if we wanted in the future. - 8501: Bug fixes for replication
- Convert
@@server_id
to uint32 value when it gets loaded as a string from config.json - Send binary logfile name in server heartbeat events, now required in MySQL 8.4
- Convert
- 8499: scanner more safe whitespace skip
- 8497: Update the protobuf version from 22.0 to 28.3
Also update README to reflect the bazel 6.3.0 version - 8493: add bats test for
kill query <pid>
- 8482: Move dolt system tables to dolt schema for doltgres
go-mysql-server
- 2727: fix dropped error in createView
- 2726: remove
resolveDescribeQuery
rule
This rule just recurses into the analyzer, so we should just do it right at the beginning, so we should do it right away and avoid calling all other rules.
This would be better/cleaner in the planbuilder, but we would need to tell the analyzer to not analyze this node or skip it somehow.
It's possible to move this logic intoanalyzeWithSelector
.
It might be more difficult to move this aboveAnalyze
, as we'd have to put this logic in a variety of places. - 2723: server/handle.go: doQuery cleanup of some error handling.
- 2722: internal/sockstate: Restore connection checking on Linux.
GMS server handler is supposed to cancel running queries if the connection which issued them goes away. It does this by checking the connection state out-of-band anytime the query is running and canceling theh query if the connection goes away. The connection checking code is platform-specific and currently only works on Linux.
In commit 538696b2c943ac7f3cacf1b67a3a5ff40ae64a00 I introduced a bug where the connection checking code tries to inspect the socket state of an already closed file descriptor. This change fixes the behavior so that the file descriptor is left open until the necessary socket state is extracted. - 2720: move
wrapTriggerRollback
logic
This PR removes theplan.TriggerRollback
andplan.NoopTriggerRollback
node and applies thetriggerRollbackIter
directly.
When there are multiple nested triggers (aka triggers that call other triggers), we used to have multiple nested trigger rollbacks with different savepoints. It turns out that was unnecessary and a single savepoint at the top of the iter tree is enough; when there's an error, it propagates back to the top, rollingback all changes. - 2718: use sqlCtx in handler
Tests here: #8493 - 2717: push
Distinct
nodes belowSort
nodes
This PR optimized queries like:When there are many duplicate values in columnselect distinct i from t order by i;
i
, it is much more efficient to eliminate duplicates first, then sort the results.
There are somewhat unrelated optimizations left as TODOs.
Optimize query: #8488
vitess
- 373: Fixed printing bug in AliasedExpr node (was calling wrong print metho…
…d, which doesn't work with custom formatters) - 371: support quoted character set values
This PR adds syntax support for quoted character set values.
fix: #8455
Closed Issues
- 8504: Crash when refreshing statistics after a PK column change
Performance
Read Tests | MySQL | Dolt | Multiple |
---|---|---|---|
covering_index_scan | 2.07 | 0.63 | 0.3 |
groupby_scan | 13.22 | 17.01 | 1.3 |
index_join | 1.34 | 2.3 | 1.7 |
index_join_scan | 1.27 | 1.89 | 1.5 |
index_scan | 34.33 | 53.85 | 1.6 |
oltp_point_select | 0.18 | 0.27 | 1.5 |
oltp_read_only | 3.49 | 5.57 | 1.6 |
select_random_points | 0.34 | 0.68 | 2.0 |
select_random_ranges | 0.39 | 0.68 | 1.7 |
table_scan | 34.33 | 54.83 | 1.6 |
types_table_scan | 75.82 | 142.39 | 1.9 |
reads_mean_multiplier | 1.5 |
Write Tests | MySQL | Dolt | Multiple |
---|---|---|---|
oltp_delete_insert | 7.98 | 5.88 | 0.7 |
oltp_insert | 3.82 | 2.91 | 0.8 |
oltp_read_write | 8.58 | 11.65 | 1.4 |
oltp_update_index | 3.89 | 2.97 | 0.8 |
oltp_update_non_index | 3.89 | 2.91 | 0.7 |
oltp_write_only | 5.37 | 5.99 | 1.1 |
types_delete_insert | 7.7 | 6.32 | 0.8 |
writes_mean_multiplier | 0.9 |
TPC-C TPS Tests | MySQL | Dolt | Multiple |
---|---|---|---|
tpcc-scale-factor-1 | 99.12 | 40.22 | 2.5 |
tpcc_tps_multiplier | 2.5 |
Overall Mean Multiple | 1.63 |
---|
1.43.9
Merged PRs
dolt
- 8494: Fix bounds check in import scanner
fixes: #8495
The before code checks whethers.i
is valid after doing the buffer access:The after code inverts the check to make surefor ; unicode.IsSpace(rune(s.buf[s.i])); s.i++ { if s.i >= s.fill { s.read() } }
s.i
is valid before indexing into the buffer.for { if s.i >= s.fill { s.read() } if !unicode.IsSpace(rune(s.buf[s.i])) { break } s.i++ }
- 8479: cross-schema foreign key support for doltgres tables
Depends on dolthub/go-mysql-server#2713
go-mysql-server
- 2717: push
Distinct
nodes belowSort
nodes
This PR optimized queries like:When there are many duplicate values in columnselect distinct i from t order by i;
i
, it is much more efficient to eliminate duplicates first, then sort the results.
There are somewhat unrelated optimizations left as TODOs.
Optimize query: #8488 - 2716: Update Parser interface documentation
Updates documentation for theParser
interface to document the requirement that implementations must returnvitess.ErrEmpty
for empty queries. Without this, GMS will not handle empty queries correctly.
Related to: dolthub/doltgresql#884 - 2715: remove exchange node
These aren't used anywhere, so it's getting removed.
Partition still exists if we ever want to reimplement some version of this. - 2713: Support for schema names in foreign key definitions
Adds schema names in many places required for foreign key operation in databases with schemas (doltgres)
Closed Issues
- 8495: Import scanner performs bound check after buffer access
Performance
Read Tests | MySQL | Dolt | Multiple |
---|---|---|---|
covering_index_scan | 2.11 | 0.62 | 0.3 |
groupby_scan | 13.7 | 17.01 | 1.2 |
index_join | 1.37 | 2.3 | 1.7 |
index_join_scan | 1.27 | 1.86 | 1.5 |
index_scan | 34.95 | 54.83 | 1.6 |
oltp_point_select | 0.18 | 0.26 | 1.4 |
oltp_read_only | 3.49 | 5.37 | 1.5 |
select_random_points | 0.34 | 0.61 | 1.8 |
select_random_ranges | 0.39 | 0.62 | 1.6 |
table_scan | 34.95 | 55.82 | 1.6 |
types_table_scan | 75.82 | 144.97 | 1.9 |
reads_mean_multiplier | 1.5 |
Write Tests | MySQL | Dolt | Multiple |
---|---|---|---|
oltp_delete_insert | 8.13 | 5.88 | 0.7 |
oltp_insert | 3.82 | 2.91 | 0.8 |
oltp_read_write | 8.58 | 11.24 | 1.3 |
oltp_update_index | 3.89 | 2.97 | 0.8 |
oltp_update_non_index | 3.89 | 2.91 | 0.7 |
oltp_write_only | 5.47 | 5.99 | 1.1 |
types_delete_insert | 7.7 | 6.32 | 0.8 |
writes_mean_multiplier | 0.9 |
TPC-C TPS Tests | MySQL | Dolt | Multiple |
---|---|---|---|
tpcc-scale-factor-1 | 99.08 | 40.84 | 2.4 |
tpcc_tps_multiplier | 2.4 |
Overall Mean Multiple | 1.60 |
---|
1.43.8
Merged PRs
dolt
- 8487: Fix server init race
Fixes a race between setting the debug log output onDoltDb
and running statistics configuration. - 8465: go/libraries/doltcore/doltdb: root_val.go: Restore ForeignKey TableNames noms serialization as just strings.
- 8453: turn on stats collect by default
- 8451: Stats purge and prune
call dolt_stats_prune()
should wipe the stats database disk contents, and replace with the currently tracked statistics. This is GC-like behavior.
call dolt_stats_purge()
should clear the stats database folder and reinitialize it empty. It's equivalent torm -rf .dolt/stats
and thendolt init
ing a new stats directory.
TODO: more testing and docs
go-mysql-server
- 2715: remove exchange node
These aren't used anywhere, so it's getting removed.
Partition still exists if we ever want to reimplement some version of this. - 2714: remove
plan.QueryProcess
and move logic tofinalizeIters
This PR removes the use ofplan.QueryProcess
and part of thetrackProcess
rule that adds/removes this node.
Instead theTrackedRowIter
is created directly infinalizeIters
. - 2710: Changing
selectExprNeedsAlias
to consider string literal quotes
When fixing a string literal quoting issue in Doltgres (dolthub/doltgresql#868), table functions stopped working, because theselectExprNeedsAlias
started returningtrue
to indicate that an alias was needed (even though it wasn't) and thebuildTableFunc
function would fail.
This started happening because the quoted string literals no longer matchedInputExpression
, sinceInputExpression
always trims off quotes when it is assigned. Another solution could be to expose thetrimQuotes
function from Vitess and use it to trimexpr.String()
before matching againstInputExpression
. - 2709: fix
LOAD DATA
64K buffer limit
This PR increase the buffer size forbufio.Scanner
toLongTextMax
, so hopefully nobody attempts to load a single row larger than 4GB.
Other changes: - 2708: removing transaction committing node
doltgres fix: dolthub/doltgresql#872 - 2702: fix: make SET system type case-insensitive
Resolves #2701 - 2697: expose planbuilder methods for domain support in Doltgres
Closed Issues
1.43.7
Merged PRs
dolt
- 8462: dolt gc --full: Implement a flag for GC which will collect everything, including the old gen.
Dolt GC is generational. By default, it moves all chunks reachable from commits into the old gen, and it leaves uncommitted data in the new gen. By default, it never revisits the chunks in the old gen. This means that if a branch with a lot of unique data exists during a GC, and then gets deleted, the storage taken up by that branch is never reclaimed by future garbage collection.
This PR implementsdolt gc --full
, which performs a collection on the entire database, ignoring the generational aspects of previous collections and visiting the entire set of live data. At the end of a successfuldolt gc --full
, only data which was reachable at the start of the run and data which has been written during the run will be present in the database.
Note:dolt gc --full
visits every reachable chunk in the database and can be resource intensive. As a consequence of runningdolt gc --full
, any previously archived table files will be unarchived, and storage savings from the archiving will be lost.dolt gc --full
involves copying chunks into new storage files, and fully materializing the new files before the old files are deleted; as a consequence, peak disk utilisation while runningdolt gc --full
can be up to 2x the existing size of the database. - 8461: Serialize schema names in foreign key constraints
- 8459: Fix merge bugs for doltgres
Closed Issues
- 7873: Running sql-server from an empty state make inconsistent repository
Performance
Read Tests | MySQL | Dolt | Multiple |
---|---|---|---|
covering_index_scan | 2.11 | 0.62 | 0.3 |
groupby_scan | 13.46 | 16.71 | 1.2 |
index_join | 1.37 | 2.3 | 1.7 |
index_join_scan | 1.27 | 1.86 | 1.5 |
index_scan | 34.33 | 54.83 | 1.6 |
oltp_point_select | 0.18 | 0.27 | 1.5 |
oltp_read_only | 3.49 | 5.47 | 1.6 |
select_random_points | 0.34 | 0.61 | 1.8 |
select_random_ranges | 0.39 | 0.62 | 1.6 |
table_scan | 34.33 | 55.82 | 1.6 |
types_table_scan | 74.46 | 147.61 | 2.0 |
reads_mean_multiplier | 1.5 |
Write Tests | MySQL | Dolt | Multiple |
---|---|---|---|
oltp_delete_insert | 8.13 | 5.88 | 0.7 |
oltp_insert | 3.82 | 2.91 | 0.8 |
oltp_read_write | 8.58 | 11.24 | 1.3 |
oltp_update_index | 3.89 | 2.97 | 0.8 |
oltp_update_non_index | 3.89 | 2.91 | 0.7 |
oltp_write_only | 5.47 | 5.88 | 1.1 |
types_delete_insert | 7.7 | 6.32 | 0.8 |
writes_mean_multiplier | 0.9 |
TPC-C TPS Tests | MySQL | Dolt | Multiple |
---|---|---|---|
tpcc-scale-factor-1 | 98.02 | 40.45 | 2.4 |
tpcc_tps_multiplier | 2.4 |
Overall Mean Multiple | 1.60 |
---|