0.16.1

This release contains a bug fix to the SQL server implementation that may cause the server to have issues with starting.

Merged PRs

• 607: Jenkinsfile: Enable AWS remote bats tests in Jenkins.
• 605: {bats,libraries/doltcore/env}: Fix init stomp bug
  Dont stomp/save any docs if one or more docs already exist in repo
• 604: Skipped bats tests for dolt init stomping existing LICENSE.md and REA…
  …DME.md
• 603: Schema export specification in bats tests
• 600: Added test for bad error message on only passing one argument to dolt…
  … push
• 599: Fixed bug for multi-key indexes
• 596: go/store/config: config_test.go: Fix test that fails when running test suite as root.
  This can happen when building and running in a golang docker container, for
  example.
• 595: [WIP] Writable Branches
  Still a WIP. Writing additional bats and unit tests.
• 594: Tim/bats schema import tags
  Skipped bats test for two sequential schema imports causing a guaranteed tag collision
• 592: /benchmark/sql_regressions/DoltRegressionsJenkinsfile: Refactor sql-watchers failure email
• 590: Add json to supported output types in dolt sql --help
  Fixes #588
• 586: Andy/Init commits don't need migration
  If a repo created with an old client ( < 0.16.0) has a branch with only the init commit, newer clients will always register that repo as un-migrated.
  This change ignores init commits when checking if a repo has been migrated
• 584: Added skipped bats test for DATETIME support in schema import
• 583: go/utils/publishrelease: Run the builds in a docker container to a get managed toolchain.

Closed Issues

• 598: enum types can't be inserted
• 589: Incorrect sql command help

0.16.0 [ACTION REQUIRED]

Dolt 0.16.0 is a very exciting release. It contains an important change to how we store columns, as well as a host of exciting features. The change to how we store columns does require users to migrate their repositories when they upgrade. We will provide some background, as well as the (very simple) migration procedure, before discussing features in the release.

We are absolutely committed to making this as painless as possible for our users, as a consequence don't hesitate to shoot us a note at [email protected] if you face any difficulty with the migration, or just need to discuss it further due to the sensitive nature of your data.

Unique Tags Migration

Dolt uses an integer value called a tag to identify columns. This contrived example illustrates that:

$ dolt init
$ cat > text.csv
name,id
novak,1
$ dolt table import -c --pk=id peeps text.csv
CREATE TABLE `peeps` (
  `name` LONGTEXT COMMENT 'tag:0',
  `id` LONGTEXT NOT NULL COMMENT 'tag:1',
  PRIMARY KEY (`id`)
);

Versions of Dolt prior to this release only required tag uniqueness per table in a given commit, not across tables and across every commit.. This caused issues when diffing and merging between commits where column tags had been reused. We decided to bite the bullet and make the fix. Going forward, all column tags will be unique across all tables and history.

Existing Dolt repositories must be migrated to be used with Dolt client 0.16.0 and later. Running a command with the new client on an old repository will result in an error message prompting you to migrate the repository. The migration process is heavily tested, deterministic, and makes no changes to underlying data. It merely fixes the format to satisfy the requirements of new versions. . After upgrading Dolt, run a single command in your repository to migrate:

$ dolt migrate

Once that's complete you should be done. If you have Dolt data living at a remote, and collaborators, there's just one additional step. The first user to upgrade will need to run:

$ dolt migrate --push

This will force push the migrated branches. Subsequent collaborators will have to then run:

$ dolt migrate --pull

This will sync their migrated repo with the migrated remote, and preserve any local changes they have, applying them on top.

SQL

We are committed to making our SQL implementation as close as possible 100% correct, and this release represents a big step towards that goal. The improvements include:

• SHOW CREATE VIEW now works, so views can be inspected in the canonical manner
• views now appear in SHOW TABLES statements
• added support for new types: DECIMAL, TIME, SET, ENUM, NCHAR, NVARCHAR, and aliases to many more
• dolt sql-server and dolt sql now support accessing multiple dolt repositories in a single SQL session. Each repository is exposed as a database. See databases with SHOW DATABASES, and select one to query with USE $database. Joins across repositories are supported. Start dolt sql or dolt sql-server with the new --multi-db-dir argument, which must name a directory containing dolt repositories to expose in queries.
• dolt sql-server now supports writes, which means that the working set will be updated by UPDATE, INSERT, DELETE and other statements which change data. The SQL server was previously read-only. This fixes #549. Important caveat: only one concurrent connection is allowed to prevent data races. Support for better concurrency is being tracked in #579
• functions user(), left(), if() are now supported, motivated by getting Dolt working with DataGrip
• saved queries now execute at the command line with dolt sql -x option
• more complete implementation of information_schema database
• JSON output option for SQL query results with dolt sql -r json

VCS in SQL

As well as making our SQL implementation compliant with MySQL, we are also committed to implementing all VCS operations available on the command line available via the SQL interface.

• we now have a dolt_branch system table where the list of branches on a repo is surfaced in SQL

Remotes

We have now fixed AWS S3 remotes, so if you want to use your own S3 infrastructure as a backend, you can do that. See the dolt remote CLI documentation for details. While we love to see data and users on DoltHub, we are committed to making Dolt and open standard, and that means being useful to the broadest possible audience.

Bug Fixes etc.

As well as fixing the issue with remotes, we fixed a number of other bugs:

• checking out or merging working set doc files
• Better SQL error messages
• SQL queries respect case of column aliases, issue here
• Queries required by Jetbrains DataGrip are now supported issue here.

Merged PRs

• 577: streaming map edits
• 576: Help Fix
  As identified by Asgavar in #553 there is a segfault caused by differences in logic between isHelp and the Parse function of the ArgParser. I found that changing the Parser to be like the isHelp function caused issues for some commands if you have a branch named help or a table named help. As a result I opted to change the isHelp logic instead.
  Thank you @Asgavar
• 572: More Types V2
  Have fun @zachmu
• 571: Skipping git-dolt bats tests on Windows due to flakiness
  Is this fine? By putting it at the end of the setup function, it's equivalent to manually putting a skip on every test. Then whenever we fix it, we can just delete it in one place.
• 569: Andy/migrate push pull
• 568: Zachmu/sql updates
  Implemented auto-commit database flavor, and use it in SQL server. Also:
  • Fix prompt formatting for shell
  • Update result printing for non-SELECT queries
  • Use a dolt version string for SQL server
    This relies on dolthub/go-mysql-server#84. Will update go.mod once it's checked in.
• 565: SQL Reserved word and SQL Keyword in column name tests
• 564: go/libraries/doltcore/env: paths.go: Consult HOME environment variable for home location before consulting os/user.
• 563: Dockerfile: Bump golang version; use mod=readonly.
• 559: Andy/migration refactor
• 556: Basic SQL batch mode bats tests
• 554: Skipped bats test for help command segfault
• 552: using correct root
• 551: Zachmu/sql json
  Added JSON result output to sql command. Also fixed handling of NULL values in CSV output.
  This fixes #533
• 548: read only version of branches table
• 547: Km/doc checkout bug
  Taylor brought a bad docs bug to my attention. If you had a modified doc, and then dolt checkout <branch> or dolt checkout -b <branch>, your local docs would be overwritten by the working root (your changes vanish).
  The intended behavior is to keep the working set exactly as it is on dolt checkout <branch>, given there are no conflicts. Since your "working set" for docs is technically on the filesystem, and not on the roots, it was getting wiped. Now i'm pulling the unstagedDocDiffs from the filesystem, and excluding those when it comes time to saveDocsOnCheckout.
  Added bats test coverage too
• 546: /bats/1pk5col-strings.bats: Add skipped test for exporting to csv then reimporting
• 545: /benchmark/sql_regressions/DoltRegressionsJenkinsfile: Add sql watcher 3
• 544: Zachmu/bheni patch
  Fixed bug in parsing URLs of relative file paths.
• 543: bats/aws-remotes.bats: Enable test for push.
• 542: Added skipped bats test for issue 538
  #538
• 540: interface for multi-db and tests
• 539: Db/dolt harness test
  Pretty simple tests, but I think are effective. Tested against the commit that initially caused the breaks and these tests failed... should also have caught the "unable to find table errors" that occurred during the harness fixing iteration process. Feels pretty seems fine to me! LMK
• 537: Jenkinsfile: Add environment variables for running AWS remote bats tests.
• 536: bats/aws-remotes.bats: Add some smoke tests for AWS remotes interactions.
  Currently these get skipped in CI. Will follow up with CI changes after this lands as they will require some external state creation and some small infra work.
  Push is skipped here and can be unskipped after #531 lands.
  Clone is skipped here and will remain skipped until another PR fixes it. I took a pass this morning but clone logic has gotten a little hairy and I wasn't happy with the progress I was making. Going to take another pass soon.
• [535](https:...

0.15.2

We are excited to announce the release of Dolt 0.15.2.

AS OF Further Enhanced

In our last release, 0.15.1 we highlighted the ability to query AS OF a branch or commit. In this release we expand this functionality by allowing uses to query AS OF a timestamp. This represents a version of the syntax familiar to users of other SQL databases with versioned data support. Thus we allow users to treat the underling commit graph as either wall clock time or relative to a commit.

TIMESTAMP and DATETIME functions

Dolt SQL now supports TIMESTAMP and DATETIME functions, which can be used to construct a time object of the given type. DATETIME('2020-03-01') and DATETIME() will return the given time or the current time, respectively.

Other

We continue to make enhancements to our underlying SQL engine, and other assorted bug fixes.

As usual, let us know if you have any thoughts by filing an issue!

Merged PRs

• 487: fix issue with merge blowing away changes in working
  Previously we used to fail if there was anything checked out and you tried to merge. Then we went to the git model where we would allow merging as long as the changes in the working set didn't touch the same tables as the changes in the commit being merged. For fast forward merges this was stomping working table changes as we'd just set the root and not re-apply the changes to the tables that were modified in the working set.
• 485: Andy/ci compatibility
• 482: Db/bats merge stats
  One test for inaccurate merge stats, one for confusion with the checkout command.
  @timsehn how do you want checkout to work when branch name and table name are identical?
• 481: go/go.mod: Upgrade dependencies.
• 480: /go/{go.mod, go.sum}: Update go.mod github.com/liquidata-inc/sqllogictest
• 478: execute saved query
  Implements:

  dolt sql -x <saved_query_name>
  dolt sql --list-saved
  

  Changes
  • dolt sql -s <name> -q <query> now saves the query with id = name
• 476: Zachmu/as of timestamp
  Support for AS OF queries with a timestamp
• 475: bump version
• 75: Added database qualifiers to view resolution.
• 74: Zachmu/datefns
  Added support for DATETIME() and TIMESTAMP() functions
• 73: Zachmu/as of
  Support for pushing AS OF clauses down to tables in a view. Kind of janky, but works great!
• 72: update vitess dep

Dolt 0.15.1 released

We are excited to announce the the release of Dolt 0.15.1.

AS OF

We now support AS OF queries, similar to the Microsoft SQL Server implementation described here. Timestamps are not yet supported for AS OF expressions, but users can use branch names, commit hashes, and other commits specs:

SELECT name FROM persons AS OF 'add-new-employees'
SELECT * FROM persons AS OF 'hvbsl13cbi03ptft78k0pgtkgpd68ehj'
SELECT * FROM persons AS OF 'HEAD~'

These queries will retrieve rows from the table as it existed at the named revision. Different tables in a join can use different AS OF clauses:

SELECT name FROM persons as of 'HEAD~' NATURAL JOIN addresses AS OF 'HEAD~2'

Other

Elsewhere we continued to improve the performance and correctness of SQL, expand test coverage, and fix bugs.

Merged PRs

• 474: Fixed bug caused by overzealous refactoring
• 471: Zachmu/batch bug
  This fixes #467.
  The SQL script in question deleted and recreated a subset of the table, and should have resulted in no diff. Before this change, it resulted in some subset of rows being deleted. The issue was that DoltDatabase in batch mode was using mapEditor.Remove and mapEditor.Add for the same keys, which doesn't work in all (most) cases. The solution is to flush the cache before and after any non-insert statement when running in batch mode.
  This change also makes the output for batch mode more sensible.
  Brian to review, Andy and Aaron FYI.
• 469: dolt sql-server bats test support
• 468: Zachmu/as of
  Dolt support for AS OF queries.
  This won't build until dolthub/go-mysql-server#71 is merged and dependencies updated.
• 465: Fixed error where we weren'tlogging an error that resulted from flush…
  …ing a batch SQL import
• 461: add information_schema database
• 460: /go/libraries/doltcore/sqle/logictest/main/main.go: Add timeout result for sqllogictests
• 456: Andy/datetime fix
• 455: Fix the SQL
  server command by providing it the appropriate variable when instantiating SQL server object.
• 453: reorganized to create testcommands package
  This PR is simply a reorganization to allow for the testcommands package, see go/libraries/doltcore/dtestutils/testcommands/command.go.
  These sorts of end-to-end tests have been very useful so far in writing rebase and super schema, and I'd like to expand their use across the codebase.
• 452: /bats/remotes-file-system.bats: Add skipped test for failed branch deletes after fetch
• 451: /bats/remotes-file-system.bats: Add skipped bats test for fetch display branch bug
• 449: adding skipped bats tests for table merge panics
• 447: First cut at generating CLI docs for docs site
  Our CLI has help text, we also use that text to generate docs for our documentation site at (DoltHub docs)[dolthub.com/docs].
  Generating docs and command line output from the same content entails three separate concerns:
  1. the raw content
  2. modifying the content so the rendering is correct (in this case shell output and Gatsby build of .mdx files, JSX version of .md)
  3. IO for writing to console (CLI help text) and files (.md files for building docs)
     The goal of this PR is to move us towards a clean separation of these three concerns using Go templates. Specifically
  • implement helper types and methods for modifying content to render correctly in CLI output or .mdx
  • modify the raw content to the new data structures and text format that can be templated
  • update markdown generation code to simply request a template, rather than building document manually
    Future work will be to use templates for the CLI output.
• 446: go/store/nbs: s3_table_reader: Be certain to close body readers when reading from S3.
  It is necessary and correct that we close these readers. Helps with persistent
  connection reuse, accurate logging and timing metrics, timely resource
  finalization, etc.
• 443: Added a succeeding bats test for push --set-upstream.
  Could not repro panic from #442.
• 441: Added skipped bats test for update a datetime field
• 439: get rid of unique counts
• 438: Db/dolt sqllogic add version to DoltRecordResult
• 437: bumped version for release
• 436: pull stomp fix
• 71: Zachmu/as of
  AS OF implementation.
  This change also removes context.Context references from the core.go interfaces, replacing them with sql.Context to be consistent.
• 69: added conversion logic to SetField expression

Closed Issues

• 467: Running piped SQL file that drops and adds a table produces different results on subsequent runs
• 466: Repo got in broken state

Dolt 0.15.0 released

We are excited to announce the release of Dolt 0.15.0.

SQL Type System

Previously Dolt had a much narrower type system than MySQL. For ease of use reasons, we just mapped types that we did not support to their "super type", for example using the previous Dolt release:

doltsql> CREATE TABLE pet (name VARCHAR(20), owner VARCHAR(20), species VARCHAR(20), sex CHAR(1), birth DATE, death DATE, PRIMARY KEY (name));
doltsql> describe pet;
+---------+----------+------+-----+---------+-------+
| Field   | Type     | Null | Key | Default | Extra |
+---------+----------+------+-----+---------+-------+
| name    | LONGTEXT | NO   | PRI |         |       |
| owner   | LONGTEXT | YES  |     |         |       |
| species | LONGTEXT | YES  |     |         |       |
| sex     | LONGTEXT | YES  |     |         |       |
| birth   | DATETIME | YES  |     |         |       |
| death   | DATETIME | YES  |     |         |       |
+---------+----------+------+-----+---------+-------+

Using this release of Dolt, we can see that richer type choices are respected:

doltsql> CREATE TABLE pet (name VARCHAR(20), owner VARCHAR(20), species VARCHAR(20), sex CHAR(1), birth DATE, death DATE, PRIMARY KEY (name));
doltsql> describe pet;
+---------+-------------+------+-----+---------+-------+
| Field   | Type        | Null | Key | Default | Extra |
+---------+-------------+------+-----+---------+-------+
| name    | VARCHAR(20) | NO   | PRI |         |       |
| owner   | VARCHAR(20) | YES  |     |         |       |
| species | VARCHAR(20) | YES  |     |         |       |
| sex     | CHAR(1)     | YES  |     |         |       |
| birth   | DATE        | YES  |     |         |       |
| death   | DATE        | YES  |     |         |       |
+---------+-------------+------+-----+---------+-------+

We hope this makes it easier for users to use Dolt in the context of their existing data infrastructure. This change is backward compatible. So old versions of Dolt can read repos written by new versions, and vice versa. That said, you should upgrade to the latest and greatest!

Unions

We also now support unions, a powerful tool for tabulating the results of analyses produced by different queries:

doltsql> (select 1 as this) union (select 2 as this);
+------+
| this |
+------+
| 1    |
| 2    |
+------+

Other

We also improved the performance of the Dolt log table, as well as making our usual assortment of bug fixes and improvements.

Merged PRs

• 435: /go/libraries/utils/iohelp/read_test.go: Skipping TestReadWithMinThroughput due to flakiness
• 434: Added skipped divide by zero bats test
• 432: go.mod: Bump go-mysql-server to support SQL UNION.
• 426: topo sort for log
• 424: Dolt log bats test rework
• 423: /go/libraries/doltcore/sqle/logictest/main/main.go: Add withdurations option
• 422: Made a file system remotes bats test file, added some tests, and move…
  …d appropriate tests from remotes.bats. Added a skipped test in remotes.bats for dolt pull stomping a dirty working set
• 419: add user agent to grpc calls
• 418: filter commits used by history table
  replaces #409
• 417: optimize the map iterator used by the history table
• 416: bh/set-algebra
  Package setalgebra provides the ability to perform algebraic set operations on mathematical sets built directly on noms
  types. Unlike standard sets in computer science, which define a finitely sized collection of unordered unique values,
  sets in mathematics are defined as a well-defined collection of distint objects. This can include infinitely sized
  groupings such as the set of all real numbers greater than 0.
  See https://en.wikipedia.org/wiki/Set_(mathematics)
  There are 3 types of sets defined in this package: FiniteSet, Interval, and CompositeSet.
  FiniteSet is your typical computer science set representing a finite number of unique objects stored in a map. An
  example would be the set of strings {"red","blue","green"}, or the set of numbers {5, 73, 127}.
  Interval is a set which can be written as an inequality such as {n | n > 0} (set of all numbers n such that n > 0) or a
  chained comparison {n | 0.0 <= n <= 1.0 } (set of all floating point values between 0.0 and 1.0)
  CompositeSet is a set which is made up of a FiniteSet and one or more non overlapping intervals such as
  {n | n < 0 or n > 100} (set of all numbers n below 0 or greater than 100) this set contains 2 non overlapping intervals
  and an empty finite set. Alternatively {n | n < 0 or {5,10,15}} (set of all numbers n below 0 or n equal to 5, 10 or 15)
  which would be represented by one Interval and a FiniteSet containing 5,10, and 15.
  There are 2 special sets also defined in this package: EmptySet, UniversalSet.
  The EmptySet is a set that has no values in it. It has the property that when unioned with any set X, X will be the
  result, and if intersected with any set X, EmptySet will be returned.
  The UniversalSet is the set containing all values. It has the property that when unioned with any set X, UniversalSet is
  returned and when intersected with any set X, X will be returned.
• 415: publishrelease/install.sh: install -d /usr/local/bin if it does not exist.
• 414: added RepoStateReader interface
• 413: Added a couple more test cases in the limit test
• 412: Skipped bats test for DATE_ADD and DATE_SUB in the where clause
• 411: Added new test repository with TypeInfo changes
• 408: Added a group by bats test highlighting inconsistent behavior
• 406: Bumped version for release
• 404: iterate a map backward
  How much do you hate this?
• 403: Tim/dateformat bats
• 68: sql/{parse,plan}: Add support union parsing and execution.
  This is still partial. We need type coercion and schema validation to be done
  in the analysis phase.
• 67: Zachmu/datemath
  Fixed panic when using interval expressions in WHERE clauses.
• 66: Negative Numbers
  Somehow I was able to not only forget to include logic to handle negative numbers, but I forgot to also write a test for them too, and even implemented this library in dolt and didn't test for negative numbers there. I'm actually surprised. It wasn't even caught in peer review. It's the simplest things that we forget to check, and those are the ones that can cause the most havoc.

Closed Issues

• 420: Dolt log -n 10 returns wrong results. Dolt log produces correct result.

0.14.0

We are pleased to announce the release of Dolt version 0.14.0.

Query Catalog

The major new feature in this release is the query catalog, which is implemented via adding additional options to the dolt sql command. Users can now pass --save and --message options to SQL queries to save and version them at the repo level:

	-s <saved query name>, --save=<saved query name>
	  Used with --query, save the query to the query catalog with the name provided. Saved queries can be examined in the dolt_query_catalog system table.

	-m <saved query description>, --message=<saved query description>
	  Used with --query and --save, saves the query with the descriptive message given. See also --name

This will allow users to document their data with versioned SQL queries. In the future we hope to make this a validation tool.

Assorted Fixes and Improvements

We continue to make progress on our SQL implementation, both in terms of correctness and performance. For example, GREATEST() now supports timestamps. We are prioritizing correctness over performance, though we are making gains in performance.

As usual, if you see anything amiss, please don't hesitate to file an issue.

Merged PRs

• 401: Fixed arg-parsing.bats not working on Windows
• 400: Added 2 skipped argument parsing bats tests.
  We don't support Nix style argument parsing completely right now
• 399: Quote view name on create view
• 395: go/cmd/dolt/commands/sql.go: Always close row iters. Improves robustness when SQL returns an error.
• 394: Km/doc diff print bug
• 393: Bumped go-mysql-server version
  Integrates dolthub/go-mysql-server#65 into dolt
• 390: bats/sql.bats: Document some unsupported SQL features with some failing bats tests.
• 389: go.mod: Upgrade dependencies.
• 386: delete cell values on drop column
• 383: deadlock fix
• 382: README.md: Add sudo invocation to install instructions.
• 379: no whitespace column names
• 378: Bumped go-mysql-server version
• 376: Added skipped test for select as
• 375: Zachmu/query save feature
  Implemented query saving via new dolt_query_catalog table, created by dolt sql -q -s.
  Also:
  • Unified business logic for validating table names in every code path where a table is created
  • Separated out read-only, read-write, and alterable SQL tables
  • Refactored deeply nested error handling logic in mv and cp commands
  • Refactored import command to separate validation from execution logic
  • Added many tests
• 372: check for nil or empty column headers
  @timsehn do we want to allow whitespace column names?
• 371: unskip test after fixing query
• 369: fix metrics bug
• 368: chunk store metrics wrapper
• 367: Changed add all shortcut from lowercase a to uppercase A to match git.
  Added corresponding bats test.
  Fixes #345
• 366: name already existing file in error message
• 365: Bumped version for 0.13.2 release
• 364: Added a basic tests for dolt_diff_ and dolt_history_ system tables.
  Found weird behavior in dolt_diff_.
• 363: Fix error message when table doesn't exist
  Fixes #275.
• 65: Added DATETIME handling to GREATEST/LEAST functions
  Fixes #380
• 63: sql/expression/function: Add UNIX_TIMESTAMP function.
• 62: Fix stupid compilation issue
• 61: small fixes
• 60: Fixed Bit type
• 59: Fixed a bug in drop table logic caused by variable shadowing. No test…
  …s of this behavior because we don't have an easy way to make an operation like DropTable return an error in the in-memory database.

Closed Issues

• 391: IGNORE
• 388: dolt diff printing bug for doc diffs
• 381: Installation problems with macOS Catalina
• 380: SQL: greatest function does not work for datetime type
• 370: Nil reference exception importing what seems to be a valid CSV

0.13.2

The set of changes for this version are:

• Bug fixes and improvements
• Output query results in CSV format
• Standards compliant CSV exports

As usual, please let us know if you have any questions or concerns.

Merged PRs

• 362: Zachmu/sql csv
  CSV output for SQL
• 361: Added skipped bats test for REPLACE counting issue
• 360: Added three simple explain bats tests
• 355: Fixed dolt documentation and improved relevant bats test
• 354: fix casing issue
• 353: A start on system tables bats tests
  @bheni will finish this and unskip the tests that are specced
• 352: ls --system and --all flags
• 351: correct schema merge
• 349: go/cmd/dolt: commands/credcmds/use.go: Implement dolt creds use to select a credential.
• 348: fixed panic on merge of non-existant branch
• 347: go/cmd/dolt: commands/credcmds/new: Use a newly created credential if there is no existing selected credential.
• 344: Bump go-mysql-server
• 343: Changed the install instructions on dolt README.
  Install instructions now point to new install scripts.
• 342: Zachmu/fix clone empty
  Cloning an empty repo now works as expected. Fixes #217
• 341: Bumping version on tip of master to fix shell script
• 339: Added tags tests
• 338: fixed conditional that was preventing batching
  Batching happens in two places: on initialization and each time the current batch is exhausted. Incorrect logic on lines 53 & 54 was preventing batching once the initial batch was exhausted.
• 337: bats,{go/cmd/dolt/commands}: Hide doc schema for schema show w/no table args
• 57: Added ChangeCollation to StringType
  Also renamed CreateBlob and MustCreateBlob to CreateBinary and MustCreateBinary because those names make more sense.

Closed Issues

• 358: Can't clone a public repo from DoltHub when not logged in.
• 340: Install Process Failing

0.13.1

We are releasing a patch fix to Dolt, as the 0.13.0 release contained a bug. The new feature for creating license and read me documents on a repository contained a bug such that when cloning a repository using dolt clone, the documents were not updated. This patch ensures that functionality works correctly.

Since this is a patch of a recent release, see 0.13.0 release notes for details about the new features recently introduced.

Merged PRs

• 335: Km/doc clone bug
• 333: fix regex

0.13.0

We are excited to announce the release of Dolt 0.13.0, hot on the heels of relaunching DoltHub.

Easy Install Script

It's now incredibly easy to install Dolt, so if you haven't tried it yet, you can now obtain a copy with a single command, and start playing with datasets:

$ curl -L https://github.com/liquidata-inc/dolt/releases/latest/download/install.sh | bash

The installer script works on Mac and Linux. For Windows, download the MSI installer below.

System Tables

We released a blog post detailing some exciting new functionality for surfacing versioning data in Dolt. This is the first of a set of features that will eventually expose all the Git-like internals of Dolt to SQL, and facilitate automated use of Dolt by allowing users to define their default choices inside SQL statements.

• dolt_log: Access the same information as the dolt log command via a SQL query
• dolt_diff_$table: A system table for each of your tables, which lets you query the diff between two commits. See the blog post for more details.
• dolt_history_$table: A system table for each of your tables, which lets you query past values of rows in the table at any commit in its history. See the blog post for more details.

LICENSE and README functionality

We now allow users to create License and Readme documents as part of their Dolt repository, these appear as LICENSE.md and README.md files in the root of your repo. Edit them with the text editor of your choice, then add them to a commit with dolt add, the same as a table. Their contents are versioned alongside your tables' data. License and Readme files will soon be visible on DoltHub for repositories that provide them. Allowing users to specify the terms on which data is available is an important step towards creating a vibrant data-sharing community.

Views

Our SQL implementation now supports persistent views, taking us closer to having a fully functioning SQL engine. Create a view using the standard SQL syntax:

CREATE VIEW myview AS SELECT col1, col2 FROM mytable

Then query it like any other table:

SELECT * FROM myview

Other

We made performance enhancements to SQL, including supporting indexed joins on a table's primary key columns. This should make the engine usable for joins on the primary key columns of two tables. Additional improvements in join performance are in the works. We also fixed assorted bugs and made performance improvements in other areas of the SQL engine.

Merged PRs

• 331: Removed Windows carriage-return and trailing whitespace from bats tests
• 329: CSV export compliant with RFC 4180
• 328: bats/helper/windows-compat.bash: Try mktemp on Windows.
• 326: one down
  The other 32 skipped bats tests are confirmed to fail
• 324: Removed old table and schema commands from the command line
• 323: fix buffered sequence iterator and put it back in row iterator
• 322: reverting buffered iter
• 320: bats/creds.bats: Debug windows failures.
• 319: Added a bats test for committing views and referencing them later
  Added some checks for checked in views.
• 318: Added test case for dolt reset --hard on new tables
• 316: Buffered Sequence Iterator
  • Created a new interface sequenceIterator for the use case when sequenceCurosor is simply accessing elements in its sequence (ie MapIterator, SetIterator, and ListIterator)
  • Created a new buffered implementation of sequenceIterator designed by @reltuk to batch chunk fetching from the ValueStore. In use cases such as DoltHub where chunk fetching IO is slow, this will dramatically accelerate performance.
• 310: Km/non-trivial merge of master into doc feature branch
  This is just a merge from master into my doc feature branch. So you can ignore that there are many commits authored by not-me.
  I wanted to get eyes on the last commit before I merge it (d4de259). I had to remove 2 of the HasDoltPrefix checks that was breaking create-views.bats. Now i'm checking for DocTableName explicitly. I left the HasDoltPrefix function since I'm using it in the commands package, and presume we'll eventually need to use it again the sqle package.
• 308: Updated to latest go-mysql-server. Re-enabled indexes by default, and…
  … un-skipped an integration test of indexed join behavior.
• 307: go/utils/publishrelease: First pass at an install.sh
• 306: Bumped go-mysql-server version
• 305: bats/creds.bats: Some initial bats tests for dolt creds new, ls and rm.
• 302: Km/doc tests
  This PR:
  • Simplifies tests in docs.bats
  • Adds tests for some helper functions in doltdb/root_val_test.go
    Will do more testing tomorrow, but wanted to get this in
• 301: dumps docs
  This code dumps the standard command line help pages for every command that isn't hidden.
  Because we only had functions for each command it was difficult to add a new method that would be implemented for each command, so I had to refactor all of that code. The refactor makes up the bulk of the PR.
• 299: dolt checkout, and merge with dolt docs, with bats coverage
  This PR includes:
  • Fixed a bug where dEnv.Docs was not always matching the docs of the current repo state (working root). This required changing the Docs type in the env package to []doltdb.DocDetails from []*doltdb.DocDetails. You'll see some reformatting to accommodate this change.
  • checkout <doc>
  • checkout <branch>
  • merge <branch> (one scenario is still buggy, need help identifying solution)
  • FF merge - docs on the FS get updated to target branch
  • Merge with conflicts - docs on the FS remain as is
  • Merge auto resolved conflicts (currently buggy) - docs on the FS should be updated to targetBranch, but they should not be added to the new working root. This would allow dolt status to indicate that the doc needs to be added and committed to finish merging. Right now it appears the doc is getting added to the working root.
• 298: go/cmd/dolt/commands/sql: Add view persistence into dolt database.
• 296: go/cmd/dolt: credcmds/check: Add dolt creds check command.
• 295: update go-mysql-server to be the latest from liquidata-inc/go-mysql-s…
  …erver@ld-master
• 294: Added indexes to dolt sqllogictest harness and updated dependency on …
  …go-mysql-server.
• 293: go/cmd/dolt/commands/credcmds: Add documentation and a little bit of chrome to dolt creds commands.
• 291: Fixed ignoring an error in put-row
• 290: go/go.mod: Run go get -u all. Migrate to dbr/v2.
• 289: Tim/add docs bats
  This is the test for branch, merge, and conflict resolve behavior. You can break it into multiple tests if you want but I think this is fine.
• 288: add diff_type column to be able to select where diff_type is added, r…
  …emoved, or modified
• 287: fixes casing issue with system tables
• 285: Added bad describe bats test per testing session with Katie
• 284: {go,bats}: Implement dolt diff by parsing docs from args, with …
  …bats test
• 283: Zachmu/explain
  Fixed describe table statements, and unskipped related tests.
• 282: change the date field to be a Sql.DateTime
  Output of the date field was in a format that wasn't able to be sorted properly.
• 281: fix select on system table that doesn't exist
  fix select on a system table that has a valid prefix but whose suffix does not match a valid table.
  What makes this a little bit tough is that you can query diffs or the history of a table that no longer exists. So need to process the entire history and then see if at any time there was a schema'd table with the given name.
• 280: {bats, go/libraries/doltcore/sqle/database.go}: Remove DoltNamespace from dolt sql command
• 279: {go,bats}: Remove DocTableName from dolt table, schema, ls, add, reset, diff
  This PR removes DocTableName from the outstanding commands so we don't expose the dolt docs table.
• 278: {go,bats}: Add dolt docs to dolt diff
  This PR adds docs to the dolt di...

0.12.0

We are excited to announce the release of Dolt 0.12.0!

Community

We have our first open-source committer to the Dolt project! Thanks to @namdnguyen for providing a helpful fix to our documentation. We are hoping this will be the first of many open-source contributions to Dolt.

SQL

As discussed in this blog post, we use sqllogictest to test our SQL implementation's logical correctness. It contains 5 million tests! This release marks a huge jump in compliance, with our implementation now hitting 89%, up from well under 50% just a few weeks ago.

Diff With Predicate

--diff-where command allows the user to add a predicate on the table being diff'd to reduce the surface area of the diff output and drill into specific data of interest.

Override Commit Date

When a user commits data, a timestamp is associated with the commit. By allowing Dolt users to customize the timestamp we allow the user to create an implicit bi-temporal database (based on commit time) while maintaining the ordinal integrity of the commit graph for querying history and reasoning about the sequence of updates.

SQL Diffs

Using the SQL diff command, that is dolt diff -q or dolt diff --sql, users can produce SQL output that will transform one branch into another. In other words this command will produce the difference, in data and schema transformations, between two refspecs in the commit log of Dolt repository.

As usual, this release also contains bug fixes and performance improvements. Please create an issue if you have any questions or find a bug.

Merged PRs

• 241: Bumped version and added release script
• 239: bats/create-views.bats: Pick up go-mysql-server support for views.
• 237: go/performance/benchmarks: remove id from results
• 236: Noticed an alter table test that now works was skipped. Unskipped.
• 235: Fix typo in README for table import
  I ran into this typo while using Dolt yesterday. The command keywords were in the incorrect order in the README.
• 233: Zachmu/sql batch
  Killed off original sql batch inserter and implemented equivalent functionality for new engine.
• 232: Andy/sqldiffrefactor
• 230: go/store/nbs: table_set.go: Rebase: Reuse upstream table file instances when supplied table specs correspond to them.
• 229: fix schema diff primary key changes
  Output looks like this for changing a pk:

  --- a/test @ 4uvb6bb3p7dqudnuidh9oh4ccsehik7n
  +++ b/test @ 2tl4quv92ot0jg4v3ai204rld00trbo4
  CREATE TABLE test (
  `pk` BIGINT NOT NULL COMMENT 'tag:0'
  -   `c1` BIGINT COMMENT 'tag:1'
  `c2` BIGINT COMMENT 'tag:2'
  `c3` BIGINT COMMENT 'tag:3'
  `c4` BIGINT COMMENT 'tag:4'
  `c5` BIGINT COMMENT 'tag:5'
  <    PRIMARY KEY (`pk`, `c1`)
  >    PRIMARY KEY (`pk`)
  );
  

  Also add the pk contstraint so it shows when it is not changing:

  --- a/test @ idfqe6c5s2i9ohihkk4r4tj70tf3l8c7
  +++ b/test @ 2tl4quv92ot0jg4v3ai204rld00trbo4
  CREATE TABLE test (
  `pk` BIGINT NOT NULL COMMENT 'tag:0'
  `c1` BIGINT COMMENT 'tag:1'
  `c2` BIGINT COMMENT 'tag:2'
  <          `c3` BIGINT COMMENT 'tag:3'
  >   `newColName3` BIGINT COMMENT 'tag:3'
  `c4` BIGINT COMMENT 'tag:4'
  `c5` BIGINT COMMENT 'tag:5'
  PRIMARY KEY (`pk`, `c1`)
  );
  

• 228: bh/add commit date
• 227: Bug fixes for sqllogictest dolt harness:
  • More inclusive types
  • Better error handling for panics
  • Cheat on tables without primary keys to allow more tests (~40%) to succeed.
• 225: disable benchmarking dolt sql imports
• 224: go/cmd/dolt: commands/sql: Keep the sql engine around throughout the lifetime of the shell / batch import.
• 221: improved super schema names
• 220: update go-mysql-server dependency
• 219: dolt benchmarking
  Initial approach is to write a script that will run n benchmarks, collect their results, then serialize those results to later be imported into dolt. Looking for feedback on approach before I head too far down this path, if it is suboptimal.
  In it's current state, there are a lot of switch statements and panics and it only accounts for types int and string and only accounts for .csv style test data formats, but I'd like to make my data generation functions robust enough to be able to account for all file formats that dolt supports and all noms types...
• 218: Added skipped bats test for schema diffs on adding a primary key
• 216: Andy/sqlschemadiffs
  Adding schema changes to dolf diff --sql output. Supports:
  • add/drop table
  • add/drop column
  • rename table
  • rename column
• 215: diff table
• 214: Bh/super schema
• 213: Zachmu/sql performance
• 212: Zachmu/sql indexes2
• 211: Added time to the handled cases in DATETIME & changed tests
  This won't compile until dolthub/go-mysql-server#26 is referenced in go.mod.