Run jetpants without any parameters to see a list of all commands available in the suite.
Please note that all Jetpants commands that take a database node require you to specify it by IP address, rather than hostname. It’s usually a good practice to list databases in your application configuration by IP anyway, to remove DNS as a potential bottleneck and single point of failure. By also keeping all commands in Jetpants by IP, it’s easier to confirm that your commands had the expected results by looking at the diff of your regenerated application config file, for example.
Here’s a more thorough description of the commands, grouped by function:
jetpants clone_slave copies data from a standby slave onto one or more fresh nodes, turning them into additional standby slaves of the same master.
Jetpants copies data sets by shutting down the MySQL daemon on the source node and then copying all MySQL files. This is the fastest way to clone MySQL data sets, and is part of the reason why we recommend having 2 standby slaves per pool for high availability.
The copy method in Jetpants uses a combination of tar, netcat (nc), and whichever compression binary you have specified in your Jetpants configuration file (if any). It does not use encryption; we assume you are transferring over a secure local network. When copying to multiple destinations, Jetpants creates a “copy chain” using tee and a fifo. For more information on this technique, please see our post on the Tumblr engineering blog.
This command does not require an asset tracker plugin, but DOES require that all your database nodes have installed whichever compression binary you specified in the Jetpants config file.
If you are using an asset tracker, when prompted for which nodes to clone TO, you may type “spare” (or equivalently just hit ENTER without typing any input) to claim a spare node with role STANDBY_SLAVE. You may clone to multiple spares at once by supplying comma-separated input like “spare, spare, spare”. You can mix-and-match with supplying IP addresses of particular hosts as well.
These commands change the type of a slave, or promote a slave to be a master. jetpants promotion does not require an asset tracker, but the other commands all do.
jetpants promotion changes which node in a pool is the master by performing a full MySQL master promotion. This is usable even if the old master is offline or unavailable. All nodes in the pool will now slave off of the new master. If the old master is online/available, it will become a standby slave of the new master.
Please note that the master promotion process enables global READ_ONLY mode on the old master. This is a required step of the most generic MySQL master promotion technique. After doing a promotion in Jetpants, you’ll need to update/deploy your application’s configuration as quickly as possible, if a plugin doesn’t do it automatically for you.
Be aware that if the old master is offline/unavailable and the pool’s slaves have replicated different amounts of data (ie, their relay logs progressed to different points at the exact moment the old master died), jetpants promotion may result in minor data inconsistencies (a couple seconds of writes) because these lost transactions are not automatically replayed on slaves that missed them. You can manually replay them using mysqlbinlog; this process is difficult to automate, which is why Jetpants and many other promotion tools do not do so. This may be implemented in a future release.
jetpants activate_slave turns a standby slave into an active slave. Use this if you want to generate an app configuration that now sends read queries to a slave that formerly did not receive them.
jetpants weigh_slave alters the weight of an active slave. This is only useful if application supports having read slaves with different selection weights.
jetpants pull_slave turns an active slave into a standby slave. Use this if you want to generate an app configuration file that stops sending read queries to a slave.
jetpants destroy_slave removes a standby slave from its pool entirely, stopping and then resetting replication on the node. Use this prior to recycling or canceling a node that is no longer needed.
jetpants console starts an irb session that is already in the Jetpants module namespace.
If you’re using an asset tracker, this is an easy way to programmatically interact with your database topology. This is especially true because the Jetpants module delegates missing methods to its Jetpants::Topology singleton. For example:
# print summary data on all pools
> pools.each &:summary
# print condensed info on shard pools only
> shards.each {|s| print "[%-12s] %8s to %-11s = %-4s\n" % [s.ip, s.min_id, s.max_id, s.data_set_size(true)]}
The first time you install Jetpants and an asset tracker, you can enter your pool information through the console:
> p = Pool.new('my-pool-name', '1.2.3.4') # supply name and master IP
> p.sync_configuration # update the asset tracker data
Or if you’re deploying a brand new pool in an existing topology:
> master = claim_spare
> slaves = claim_spares(2)
> p = Pool.new('my-new-pool', master)
> slaves.each {|s| s.change_master_to(master); s.start_replication}
> p.sync_configuration
Jetpants allows you to split a shard into N pieces. This is a fairly complex process; the steps look roughly like this:
-
Create N new slaves of the shard being split (the “parent” shard). These new slaves will become masters of their own “child” shards later in the process.
-
Reduce the data set on those slaves so that each contains a different subset of the data. This is done by exporting the data subset, dropping the tables, re-creating the tables, and importing the data subset back in.
-
Move app reads from the parent shard to the appropriate child shards. You must move reads before moving writes in order to maintain consistency in your application. Writes will continue to go to the parent shard, which then replicates to the child shards.
-
Move app writes from the parent shard to the appropriate child shards.
-
Stop replicating writes from the parent shard, and then take the parent pool offline entirely.
-
Remove rows that replicated to the wrong child shard. This data will be sparse, since it’s only the writes that were made since the shard split process started.
For more information, including diagrams of each step, please see our presentation at Percona Live 2013.
Separately, Jetpants also allows you to alter the range of the last shard in your topology. In a range-based sharding scheme, the last shard has a range of X to infinity; eventually this will be too large of a range, so you need to truncate that shard range and create a “new” last shard after it. We call this process “shard cutover”.
These commands all require an asset tracker in order to function.
jetpants shard_split performs the first steps of the split: spins up the new shard pools with the appropriate portions of the data set. The new shards will still be slaving from their parent though. This step potentially takes several hours, so you may want to use a tool like screen to prevent SSH timeouts.
jetpants shard_split_child_reads regenerates your application config file to move reads to the new child shards.
jetpants shard_split_child_writes regenerates your application config file to move writes to the new child shards. Be sure to move reads first.
jetpants shard_split_cleanup tears down replication between the parent and child shards, makes the parent shard non-readable and non-writable, and then cleans up data that replicated to the wrong child. This may take a little while to run, although typically much less than the initial jetpants shard_split takes since data will be sparse.
jetpants shard_cutover truncates the current last shard range, and adds a new shard after it. You need to pick what ID is being used for the cutover; this ID MUST be one “in the future”, ie, one your application hasn’t hit yet. The cutover process does NOT move any data, since it assumes the new shard it’s creating has no data yet. What the cutover process will do is allocate the new shard’s hardware, set up replication, create your sharded tables, and update your app configuration file.
With an asset tracker, Jetpants allows you to mark a shard as read-only or completely offline, assuming your application supports these notions. (Which, of course, it should – this is one of the major benefits of a sharded infrastructure!)
jetpants shard_read_only marks a shard as not writable.
jetpants shard_offline marks a shard as completely unavailable.
jetpants shard_online marks a shard as being fully online again, after a prior call to jetpants shard_read_only or jetpants shard_offline.
These commands display status information about a particular node, pool, or the entire topology.
jetpants summary displays information about a node, along with its pool. Does not require an asset tracker.
jetpants pools displays full information about all pools (name, size, full node list including roles), and then displays counts of all nodes by role. Requires an asset tracker to obtain the list of all pools. This command may take a minute or two to run (depending on topology size) since it has to probe every node to determine roles.
jetpants pools_compact displays condensed information about all pools (name, master IP, size). Requires an asset tracker to obtain the list of all pools.
jetpants defrag_slave exports all data on a standby slave, drops tables, recreates tables, and then re-imports the data. This is useful for defragmenting a node. Currently it only works on shard pool slaves; this may change in a future release.
jetpants regen_config regenerates your application’s configuration file, assuming you are using an asset tracker. This isn’t terribly useful in most cases, because any commands that make configuration changes will do this automatically. However, you may want to run this after manually making topology changes via jetpants console.