Skip to content
forked from mongock/mongock

Super lightweight MongoDB migration tool for Java

License

Notifications You must be signed in to change notification settings

ksa-real/mongomig

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Maven Central Build Status Bugs Vulnerabilities Hex.pm

LAST NEWS ‼️‼️💥💥

Bug found in SpringBoot module in all versions up to 2.0.1. Use of SpringBoot module not recommended until fix

Mongock: MongoDB version control tool for Java

mongock is a java MongoDB tool for tracking, managing and applying database schema changes.

The motivation of this project is to add some important and useful features, provide a different code design, which we believe is easier to maintain, extend and debug, applying what we believe are best practices, and specially providing a more fluent maintenance to apply collaborator's contributions. The concept is very similar to other db migration tools such as Liquibase or Flyway but without using XML/JSON/YML files.

mongock provides new approach for adding changes (change sets) based on Java classes and methods with appropriate annotations.

Table of contents

Why Mongock

There are several good reasons to use Mongock in your project. Here we give you some of them:

  • Solid solution which really works.
  • Works well with sharded collections: Unlike other similar projects using javascript, which requires db.eval(). Documentation.
  • Distributed solution with solid locking mechanism.
  • We are very responsive, aiming for 24-hours-response for new issues and 48 hours for reviews(Notice this is not a commitment, but so far so good).
  • Well maintained and regularly updated.
  • Used by several tech companies in different industries.
  • Can be used together with most, if not all, frameworks.
  • Provides great integration for Spring, allowing you to inject any dependency you want to your changelog method.

Sample projects

In here you can find some sample projects that show you how to use Mongock.

Contributing

If you would like to contribute to Mongock project, please read how to contribute for details on our collaboration process and standards.

Add a dependency

Mongock can be used standalone, with Spring, or with Jongo. The mongock-core dependency is always required, and either mongock-spring or mongock-jongo can also be added. Using mongock-spring with mongock-jongo is not currently supported.

With Maven

<!-- To use standalone (i.e., w/o Spring or Jongo) -->
<dependency>
  <groupId>com.github.cloudyrock.mongock</groupId>
  <artifactId>mongock-core</artifactId>
  <version>2.0.0</version>
</dependency>

<!-- Add to use with Spring-->
<dependency>
  <groupId>com.github.cloudyrock.mongock</groupId>
  <artifactId>mongock-spring</artifactId>
  <version>2.0.0</version>
</dependency>


<!-- Add to use with Jongo -->
<dependency>
  <groupId>com.github.cloudyrock.mongock</groupId>
  <artifactId>mongock-jongo</artifactId>
  <version>2.0.0</version>
</dependency>

With Gradle

compile 'org.javassist:javassist:3.18.2-GA'          // workround for ${javassist.version} placeholder issue*
compile 'com.github.cloudyrock.mongock:mongock-core:2.0.0'    // standalone
compile 'com.github.cloudyrock.mongock:mongock-spring:2.0.0'  // with Spring (in addition to mongock-core)
compile 'com.github.cloudyrock.mongock:mongock-jongo:2.0.0'   // with Jongo (in addition to mongock-core

Usage with Spring...Mongock as a Bean

You need to instantiate mongock object and provide some configuration. If you use Spring, mongock can be instantiated as a singleton bean in the Spring context. In this case the migration process will be executed automatically on startup.

@Bean
public SpringMongock mongock() {
  MongoClient mongoclient = new MongoClient(new MongoClientURI("yourDbName", yourMongoClientBuilder));
  return new SpringMongockBuilder(mongoclient, "yourDbName", "com.package.to.be.scanned.for.changesets")
      .setLockQuickConfig()
      .build();
}

Usage with SpringBoot...When you need to inject your own dependencies

The main benefit of using SpringBoot integration is that it provides a totally flexible way to inject dependencies, so you can inject any object to your change logs by using SpringBoot ApplicationContext.

In order to use this feature you need to instantiate the SpringBoot mongock class and provide the required configuration. Mongock will run as an ApplictionRunner within SpringBoot. In terms of execution, it will be be very similar to the standard Spring implementation. The key difference is that ApplicationRunner beans run after (as opposed to during) the context is fully initialized.

Note: Using this implementation means you need all the dependencies in your changelogs(parameters in methods annotated with @ChangeSet) declared as Spring beans.

@Bean
public SpringBootMongock mongock(ApplicationContext springContext, MongoClient mongoClient) {
  return new SpringBootMongockBuilder(mongoClient, "yourDbName", "com.package.to.be.scanned.for.changesets")
      .setApplicationContext(springContext) 
      .setLockQuickConfig()
      .build();
}

Usage with Jongo

Using mongock with Jongo is similar, but you have to remember to run execute to start the migration process.

  MongoClient mongoclient = new MongoClient(new MongoClientURI("yourDbName", yourMongoClientBuilder));
  JongoMongock runner=  new JongoMongockBuilder(mongoclient, "yourDbName", "com.package.to.be.scanned.for.changesets")
      .setJongo(myJongo) 
      .setLockQuickConfig()
      .build();
  runner.execute();         //  ------> starts migration changesets

Standalone usage

Using mongock standalone is similar to with Jongo.

  MongoClient mongoclient = new MongoClient(new MongoClientURI("yourDbName", yourMongoClientBuilder));
  Mongock runner=  new MongockBuilder(mongoclient, "yourDbName", "com.package.to.be.scanned.for.changesets")
      .setLockQuickConfig()
      .build();
  runner.execute();         //  ------> starts migration changesets

Above examples provide minimal configuration. The various Mongock builders provide some other possibilities (setters) to make the tool more flexible:

builder.setChangelogCollectionName(logColName);   // default is dbchangelog, collection with applied change sets
builder.setLockCollectionName(lockColName);       // default is mongocklock, collection used during migration process
builder.setEnabled(shouldBeEnabled);              // default is true, migration won't start if set to false

More about URI

Creating change logs

ChangeLog contains bunch of ChangeSets. ChangeSet is a single task (set of instructions made on a database). In other words ChangeLog is a class annotated with @ChangeLog and containing methods annotated with @ChangeSet.

package com.example.yourapp.changelogs;

@ChangeLog
public class DatabaseChangelog {
  
  @ChangeSet(order = "001", id = "someChangeId", author = "testAuthor")
  public void importantWorkToDo(DB db){
     // task implementation
  }


}

@ChangeLog

Class with change sets must be annotated by @ChangeLog. There can be more than one change log class but in that case order argument should be provided:

@ChangeLog(order = "001")
public class DatabaseChangelog {
  //...
}

ChangeLogs are sorted alphabetically by order argument and changesets are applied due to this order.

@ChangeSet

Method annotated by @ChangeSet is taken and applied to the database. History of applied change sets is stored in a collection called dbchangelog (by default) in your MongoDB

Annotation parameters:

order - string for sorting change sets in one changelog. Sorting in alphabetical order, ascending. It can be a number, a date etc.

id - name of a change set, must be unique for all change logs in a database

author - author of a change set

runAlways - [optional, default: false] changeset will always be executed but only first execution event will be stored in dbchangelog collection

Defining ChangeSet methods

Method annotated by @ChangeSet can have one of the following definition:

@ChangeSet(order = "001", id = "someChangeWithoutArgs", author = "testAuthor")
public void someChange1() {
   // method without arguments can do some non-db changes
}

@ChangeSet(order = "002", id = "someChangeWithMongoDatabase", author = "testAuthor")
public void someChange2(MongoDatabase db) {
  // type: com.mongodb.client.MongoDatabase : original MongoDB driver v. 3.x, operations allowed by driver are possible
  // example: 
  MongoCollection<Document> mycollection = db.getCollection("mycollection");
  Document doc = new Document("testName", "example").append("test", "1");
  mycollection.insertOne(doc);
}

@ChangeSet(order = "003", id = "someChangeWithDb", author = "testAuthor")
public void someChange3(DB db) {
  // This is deprecated in mongo-java-driver 3.x, use MongoDatabase instead
  // type: com.mongodb.DB : original MongoDB driver v. 2.x, operations allowed by driver are possible
  // example: 
  DBCollection mycollection = db.getCollection("mycollection");
  BasicDBObject doc = new BasicDBObject().append("test", "1");
  mycollection .insert(doc);
}

@ChangeSet(order = "004", id = "someChangeWithJongo", author = "testAuthor")
public void someChange4(Jongo jongo) {
  // type: org.jongo.Jongo : Jongo driver can be used, used for simpler notation
  // example:
  MongoCollection mycollection = jongo.getCollection("mycollection");
  mycollection.insert("{test : 1}");
}

@ChangeSet(order = "005", id = "someChangeWithSpringDataTemplate", author = "testAuthor")
public void someChange5(MongoTemplate mongoTemplate) {
  // type: org.springframework.data.mongodb.core.MongoTemplate
  // Spring Data integration allows using MongoTemplate in the ChangeSet
  // example:
  mongoTemplate.save(myEntity);
}

@ChangeSet(order = "006", id = "someChangeWithSpringDataTemplate", author = "testAuthor")
public void someChange6(MongoTemplate mongoTemplate, Environment environment) {
  // type: org.springframework.data.mongodb.core.MongoTemplate
  // type: org.springframework.core.env.Environment
  // Spring Data integration allows using MongoTemplate and Environment in the ChangeSet
}

Injecting custom dependencies to change logs

Right now this is possible by using SpringBoot Application Context. See SpringBoot set up for more information. However, this feature will be available for standalone and Jongo implementations.

Using Spring profiles

mongock accepts Spring's org.springframework.context.annotation.Profile annotation. If a change log or change set class is annotated with @Profile, then it is activated for current application profiles.

Example 1: annotated change set will be invoked for a dev profile

@Profile("dev")
@ChangeSet(author = "testuser", id = "myDevChangest", order = "01")
public void devEnvOnly(DB db){
  // ...
}

Example 2: all change sets in a changelog will be invoked for a test profile

@ChangeLog(order = "1")
@Profile("test")
public class ChangelogForTestEnv{
  @ChangeSet(author = "testuser", id = "myTestChangest", order = "01")
  public void testingEnvOnly(DB db){
    // ...
  } 
}

Enabling @Profile annotation (option)

To enable the @Profile integration, please inject org.springframework.core.env.Environment to your runner.

@Bean 
@Autowired
public SpringMongock mongock(Environment environment) {
  SpringMongock runner = new SpringMongockBuilder(mongoclient, "yourDbName", "com.package.to.be.scanned.for.changesets")
      .setSpringEnvironment(environment)
      .setLockQuickConfig()
      .build();

  //... etc
}

Configuring Lock

In order to execute the changelogs, mongock needs to manage the lock to ensure only one instance executes a changelog at a time. By default the lock is reserved 24 hours and, in case the lock is held by another mongock instance, will ignore the execution and no exception will be sent, unless the parameter throwExceptionIfCannotObtainLock is set to true.

There are 3 parameters to configure:

lockAcquiredForMinutes - Number of minutes mongock will acquire the lock for. It will refresh the lock when is close to be expired anyway.

maxTries - Max tries when the lock is held by another mongock instance.

maxWaitingForLockMinutes - Max minutes mongock will wait for the lock in every try.

To configure these parameters there are two methods: setLockConfig and setLockConfig and setLockQuickConfig. Both will set the parameter throwExceptionIfCannotObtainLock to true.

@Bean @Autowired
public Mongock mongock(Environment environment) {
  Mongock runner = new mongock(uri);
  runner.setLockConfig(5, 6, 3);
}

or quick config with 3 minutes for lockAcquiredFor, 3 max tries and 4 minutes for maxWaitingForLock

 @Bean @Autowired
 public Mongock mongock(Environment environment) {
   Mongock runner = new mongock(uri);
   runner.setLockQuickConfig();
 }

Known issues

Mongo java driver conflicts

mongock depends on mongo-java-driver. If your application has mongo-java-driver dependency too, there could be library conflicts in some cases.

Exception:

com.mongodb.WriteConcernException: { "serverUsed" : "localhost" , 
"err" : "invalid ns to index" , "code" : 10096 , "n" : 0 , 
"connectionId" : 955 , "ok" : 1.0}

Workaround:

You can exclude mongo-java-driver from mongock and use your dependency only. Maven and gradle examples below:

<!-- pom.xml -->
<dependency>
    <groupId>org.mongodb</groupId>
    <artifactId>mongo-java-driver</artifactId>
    <version>3.4.0</version>
</dependency>

<dependency>
  <groupId>com.github.cloudyrock.mongock</groupId>
  <artifactId>mongock-core</artifactId>
  <version>2.0.0</version>
  <exclusions>
    <exclusion>
      <groupId>org.mongodb</groupId>
      <artifactId>mongo-java-driver</artifactId>
    </exclusion>
  </exclusions>
</dependency>
    // build.gradle
    compile "org.mongodb:mongo-java-driver:3.4.0"
    compile("com.github.cloudyrock.mongock:mongock:2.0.0") {
        exclude group: 'org.mongodb', module: 'mongo-java-driver'
    }

Mongo transaction limitations

Due to Mongo limitations, there is no way to provide atomicity at ChangelogSet level. So a Changelog could need more than one execution to be finished, as any interruption could happen, leaving the changelog in a inconsistent state. If that happens, the next time mongock is executed it will try to finish the changelog execution, but it could already be half executed.

For this reason, the developer in charge of the changelog's design, should make sure that:

  • Changelog is idempotent: As changelog can be interrupted at any time, it will need to be executed again.
  • Changelog is Backward compatible (If high availability is required): While the migration process is taking place, the old version of the software is still running. During this time could happen(and probably will) that the old version of the software is dealing with the new version of the data. Could even happen that the data is a mix between old and new version. This means the software must still work regardless of the status of the database. In case the developer is aware of this and still decides to provide a non-backward-compatible changeSet, he should know it's a detriment to high availability.
  • Changelog reduces its execution time in every iteration: This is harder to explain. As said, a changelog can be interrupted at any time. This means an specific changelog needs to be re-run. In the undesired scenario where the changelog's execution time is grater than the interruption time(could be Kubernetes initial delay), that changelog won't be ever finished. So the changelog needs to be developed in such a way that every iteration reduces its execution time, so eventually, after some iterations, the changelog finished.
  • Changelog's execution time is shorter than interruption time: In case the previous condition cannot be ensured, could be enough if the changelog's execution time is shorter than the interruption time. This is not ideal as the execution time depends on the machine, but in most case could be enough.

Code of conduct

Please read the code of conduct for details on our code of conduct.

LICENSE

Mongock propject is licensed under the Apache License Version 2.0. See the LICENSE file for details

About

Super lightweight MongoDB migration tool for Java

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 99.2%
  • Shell 0.8%