Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ of [Vitess v2.0.0](https://github.com/youtube/vitess/releases). Some highlights
* Java and Go clients use the new HTTP/2-based [gRPC](http://www.grpc.io/) framework.
* Can now run on top of MySQL 5.6, in addition to MariaDB 10.0.
* New administrative dashboard built on AngularJS.
* Built-in backup/restore</span>](http://vitess.io/user-guide/backup-and-restore.html), designed to plug into blob stores like [Google Cloud Storage](https://cloud.google.com/storage/).
* GTID-based [reparenting](http://vitess.io/user-guide/reparenting.html) for reversible, routine failovers.
* Simpler [schema changes](http://vitess.io/user-guide/schema-management.html).
* [Built-in backup/restore</span>](https://vitess.io/docs/user-guides/operating-vitess/backup-and-restore/) designed to plug into blob stores like [Google Cloud Storage](https://cloud.google.com/storage/).
* GTID-based [reparenting](https://vitess.io/docs/user-guides/configuration-advanced/reparenting/) for reversible, routine failovers.
* Simpler [schema changes](https://vitess.io/docs/reference/features/schema-management/).

We've also been hard at work adding lots more [documentation](http://vitess.io/user-guide/introduction.html). In particular, the rest of this post will explore one of our new walkthroughs that demonstrates transparent [resharding](http://vitess.io/user-guide/sharding.html#resharding) of a live database - that is, changing the number of shards without any code changes or noticeable downtime for the application.

Expand All @@ -38,30 +38,22 @@ values, which we call [keyspace IDs](http://vitess.io/overview/concepts.html#key

**Transparent Resharding**

If you want to follow along with the new [resharding walkthrough](http://vitess.io/user-guide/sharding-kubernetes.html), you'll need to first bring up the cluster as described in the [unsharded guide](http://vitess.io/getting-started/). Both guides use the same [sample app](https://github.com/youtube/vitess/tree/master/examples/kubernetes/guestbook), which is a Guestbook that supports multiple, numbered pages.

![Guestbook page 72 screenshot](../images/2015-10-06-cloud-native-mysql-sharding-with-vitess-and-kubernetes-vitess%2B5.png)
If you want to follow along with the new [resharding walkthrough](http://vitess.io/user-guide/sharding-kubernetes.html), you'll need to first bring up the cluster as described in the [unsharded guide](http://vitess.io/getting-started/).

In the [sample app code](https://github.com/youtube/vitess/blob/master/examples/kubernetes/guestbook/main.py), you'll see a `get\_keyspace\_id()` function that transforms a given page number to the set of all 8-byte sequences, establishing the mapping we need for consistent hashing. In the unsharded case, these values are stored but not used. When we introduce sharding, page numbers will be evenly distributed (on average) across all the shards we create, allowing the app to scale to support arbitrary amounts of pages.

Before resharding, you'll see a single [custom shard](http://vitess.io/user-guide/sharding.html#custom-sharding) named "0" in the Vitess dashboard. This is what an unsharded [keyspace](http://vitess.io/overview/concepts.html#keyspace) looks like.

![Guestbook dashboard screenshot](../images/2015-10-06-cloud-native-mysql-sharding-with-vitess-and-kubernetes-vitess%2B1.png)

As you begin the [resharding walkthrough](http://vitess.io/user-guide/sharding-kubernetes.html),you'll bring up two new shards for the same keyspace. During resharding, the new shards will run alongside the old one, but they'll remain idle (Vitess will not route any app traffic to them) until you're ready to migrate. In the dashboard, you'll see all three shards, but only shard "0" is currently active.

![Guestbook test_keyspace image](../images/2015-10-06-cloud-native-mysql-sharding-with-vitess-and-kubernetes-vitess%2B2.png)

Next, you'll run a few Vitess commands to [copy the schema and data](http://vitess.io/user-guide/sharding-kubernetes.html#copy-data-from-original-shard) from the original shard. The key to live migration is that once the initial snapshot copy is done, Vitess will automatically begin replicating fresh updates on the original shard to the new shards. We call this [filtered replication](http://vitess.io/user-guide/sharding.html#filtered-replication), since it distributes DMLs only to the shards to which they apply. Vitess also includes tools that compare the original and copied data sets, row-by-row, to [verify data
integrity](http://vitess.io/user-guide/sharding-kubernetes.html#check-copied-data-integrity).

Once you've verified the copy, and filtered replication has caught up to real-time updates, you can run the [migrate command](http://vitess.io/user-guide/sharding-kubernetes.html#switch-over-to-the-new-shards) which tells Vitess to atomically shift app traffic from the old shards to the new ones. It does this by disabling writes on the old masters, waiting for the new masters to receive the last events over filtered replication, and then enabling writes on the new masters. Since the process is automated, this typically only causes about a second of write unavailability.

Now you can[tear down the old shard](http://vitess.io/user-guide/sharding-kubernetes.html#remove-the-original-shard),
Now you can [tear down the old shard](http://vitess.io/user-guide/sharding-kubernetes.html#remove-the-original-shard),
and verify that only the new ones show up in the dashboard.

![Guestbook dashboard image verify new shards](../images/2015-10-06-cloud-native-mysql-sharding-with-vitess-and-kubernetes-vitess%2B4.png)

Note that we never had to tell the app that we were changing from one shard to two. The resharding process was completely transparent to the app, since Vitess automatically reroutes queries on-the-fly as the migration progresses.

At YouTube, we've used Vitess to transparently reshard (both [horizontally and vertically](http://vitess.io/user-guide/sharding.html#supported-operations)) nearly all of our MySQL databases within the last year alone, and we have still more on the horizon as we continue to grow. See the full [walkthrough instructions](http://vitess.io/user-guide/sharding-kubernetes.html) if you want to try it out for yourself.
Expand All @@ -73,14 +65,10 @@ The promise of sharding is that it allows you to scale write throughput linearly
Below you can see preliminary results for scaling write throughput by adding more shards in Vitess running on [Google Container Engine](https://cloud.google.com/container-engine/). For this benchmark, we pointed YCSB at the [load balancer](http://kubernetes.io/v1.0/docs/user-guide/services.html#type-loadbalancer) for our Vitess cluster and told it to send a lot of INSERT statements.
Vitess took care of routing statements to the various shards.

[![](../images/thumbnails/2015-10-06-cloud-native-mysql-sharding-with-vitess-and-kubernetes-vitess%2B3.png)](../images/2015-10-06-cloud-native-mysql-sharding-with-vitess-and-kubernetes-vitess%2B3.png)

The max throughput (QPS) for a given number of shards is the point at which round-trip write latency became degraded, which we define as &gt;15ms on average or &gt;50ms for the worst 1% of queries (99th percentile).

We also ran YCSB's "read mostly" workload (95% reads, 5% writes) to show how Vitess can scale read traffic by adding replicas. The max throughput here is the point at which round-trip read latency became degraded, which we define as &gt;5ms on average or &gt;20ms for the worst 1% of queries.

![Guestbook image](../images/2015-10-06-cloud-native-mysql-sharding-with-vitess-and-kubernetes-vitess%2B6.png)

There's still a lot of room to improve the benchmarks (for example, by tuning the performance of MySQL itself). However, these preliminary results show that the returns don't diminish as you scale. And since you're scaling horizontally, you're not limited by the size of a single machine.

**Conclusion**
Expand Down
2 changes: 1 addition & 1 deletion content/en/docs/contributing/code-reviews.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ In order to avoid disruption, the following concerns need to be kept in mind:
Every GitHub pull request must go through a code review and get approved before it will be merged into the master branch.

Every pull request should meet the following requirements:
* Use the [Pull Request Template](https://github.com/vitessio/vitess/blob/master/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md)
* Use the [Pull Request Template](https://github.com/vitessio/vitess/blob/master/.github/pull_request_template.md)
* Adhere to the [Go coding guidelines](https://golang.org/doc/effective_go.html) and watch out for these [common errors](https://github.com/golang/go/wiki/CodeReviewComments).
* Contain a description message that is as detailed as possible. Here is a great example https://github.com/vitessio/vitess/pull/6543.
* Pass all CI tests that run on PRs.
Expand Down
5 changes: 3 additions & 2 deletions content/en/docs/resources/presentations.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,8 +194,9 @@ Vitess team member [Anthony Yeh](https://github.com/enisoc)'s talk at
the [January 2016 CoreOS Meetup](http://www.meetup.com/coreos/events/228233948/)
discussed challenges and techniques for running distributed databases
within Kubernetes, followed by a deep dive into the design trade-offs
of the [Vitess on Kubernetes](https://github.com/vitessio/vitess/tree/master/examples/kubernetes)
deployment templates.
of the Vitess on Kubernetes deployment templates.

Please note the Vitess on Kubernetes deployment templates were removed as of February 27, 2020.

{{< pdf src="/ViewerJS/#../files/coreos-meetup-2016-01-27.pdf" >}}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Unmanaged Schema Changes
weight: 3
aliases: ['/docs/schema-management/unmanaged-schema-changes/', '/docs/user-guides/unmanaged-schema-changes/']
aliases: ['/docs/user-guides/operating-vitess/making-schema-changes', '/docs/schema-management/unmanaged-schema-changes/', '/docs/user-guides/unmanaged-schema-changes/']
---

Vitess offers multiple approaches to running unmanaged schema changes. Below, we review each of these approaches.
Expand Down