Skip to content

API Reference

Jump to bottom
Yu Xia edited this page Oct 11, 2015 · 37 revisions

API Convention

Unless explicitly stated, all API is async and returns a bluebird promise

You can use generator on node v0.12+ with --harmony option

var conn = yield memdb.connect(opts);

Or use promise on previous node version

memdb.connect(opts).then(function(conn){
    // use conn here
});

The following is incorrect

var conn = memdb.connect(opts); //DO NOT DO THIS!

memdb binaries

memdbd

The memdb server

Usage

memdbd --shard=[shardId] [options]

Options

  • --shard(required) - shardId to start
  • --conf - path. Memdb config file path. If not specified, the following paths will be searched in order [~/.memdb/memdb.conf.js, /etc/memdb.conf.js]
  • --daemon - start as daemon
  • --help - display help

Example

memdbd --shard=s1 --conf=memdb.js --daemon

To shutdown server, just press ^C when in console mode, or send signal kill to the process when in daemon mode.

memdb.conf.js

Memdb server config file

/*
 * MemDB config template
 *
 * Please modify it on your needs
 *
 * This is plain javascript, you can add any js code here, just export the config
 */

module.exports = {
    // *** global settings for all shards ***

    // Global backend storage, all shards must connect to the same mongodb (or mongodb cluster)
    backend : {
        engine : 'mongodb', // should be 'mongodb'
        url : 'mongodb://localhost/memdb-test', // mongodb connect string
        options : {}, // mongodb connect options
    },

    // Global locking redis, all shards must connect to the same redis (or redis cluster)
    locking : {
        host : '127.0.0.1',
        port : 6379,
        db : 0,
    },

    // Data replication redis, one redis instance for each shard
    // You can override this in shard settings to choice different slave for each shard
    slave : {
        host : '127.0.0.1',
        port : 6379,
        db : 0,
    },

    // Log settings
    log : {
        // Log file path
        path : '/tmp',
        // Log Level (one of 'ALL', 'TRACE', 'DEBUG', 'INFO', 'WARN', 'ERROR', 'OFF')
        // Please set to WARN on production
        level : 'WARN',
    },

    // Promise settings
    promise : {
        // Enable long stack trace, disable it on production
        longStackTraces : false,
    },

    // user for memdbcluster ssh login, default current user
    // when start using memdbcluster, make sure you have ssh permission (without password) on all servers,
    // and the memdb version, install folder, config files are all the same in all servers
    user : process.env.USER,

    // Collection settings for index
    collections : {
        // Collection name
        player : {
            // Index setting, modify it on your need
            indexes : [
                {
                    // Index keys
                    keys : ['areaId'],
                    // Value exclude from index. Values like '', -1 occurs too often, which can make the index too large.
                    // 'null' or 'undefined' is ignored by default.
                    valueIgnore : {
                        areaId : ['', -1],
                    },
                },
                {
                    // Index keys (compound index)
                    keys : ['deviceType', 'deviceId'],
                    // Unique constraint for the index
                    unique : true,
                },
            ]
        }
    },


    // *** Shard specific settings ***

    // This will override global settings on specifid shard
    shards : {
        // shardId
        s1 : {
            // server IP
            host : '127.0.0.1',
            // listen port
            port : 31017,
            // bind Ip
            // DO NOT bind to localhost when deploy on multiple servers
            // make sure servers can communicate with each other            
            bind : '0.0.0.0',

            // Add any shard specific settings here
            // slave : {
            //     host : '127.0.0.1',
            //     port : 6379,
            //     db : 1,
            // },
        },

        // Add more shards
        s2 : {
            host : '127.0.0.1',
            port : 31018,
        },
    }


    // *** additional settings ***
    // These settings are unstable and may change in later version

    // Delay for flush changes to backend storage
    // Set it to large value to improve performance if the data delay in backend storage is not an issue.
    // persistentDelay : 600 * 1000, // number in ms, default 10 min. 0 indicates never

    // Idle time before document is removed from memory.
    // Larger value can improve performance but use more memory.
    // Set it to large value if the documents accessed via this shard is limited.
    // Do not access too many different documents in a short time, which may exhault memory and trigger heavy GC operation.
    // idleTimeout : 1800 * 1000, // number in ms, default 30 min. 0 indicates never

    // GC will be triggered when memory usage reach this limit
    // GC can be very heavy, please adjust idleTimeout to avoid GC.
    // memoryLimit : 1024, // number in MB, default 1024

    // Disable redis replica, DO NOT turn on this in production.
    // disableSlave : false, // default false
    
    // Slow query time
    // slowQuery : 2000, // number in ms. default 2000

    // Turn on heapdump module (https://www.npmjs.com/package/heapdump)
    // heapdump : false, // default false    
};

memdb

Memdb interactive shell

The shell is just a nodejs repl, you can use all features provided by nodejs repl

Usage:

memdb [options]

Options:

  • -h, --host - ip. shard ip
  • -p, --port - port. shard port
  • -s, --shard - shardId.
  • -c, --conf - path. config file path
  • -h, --help display help

Example:

memdb -h127.0.0.1 -p31017
memdb -s s1 -c memdb.conf.js

Play with shell

memdb> db.insert('player', {_id : 1, name : 'rain'}) // insert a doc to 'player' collection
'1'
memdb> db.find('player', 1)  // find doc by id
{ _id: '1', name: 'rain' }
memdb> db.commit() // commit changes
true
memdb> db.update('player', 1, {$set : {name : 'snow'}}) // update doc
1
memdb> db.find('player', 1, 'name')
{ name: 'snow' }
memdb> db.rollback() // rollback changes
true
memdb> db.find('player', 1, 'name')
{ name: 'rain' }
memdb> db.remove('player', 1) // remove doc
1
memdb> db.commit()
true
memdb> db.info()
{
    connId: '5e70898b-7349-4a05-8e0f-42fcad6a91e4', // connection id
    ver: '0.4.0', // server version
    uptime: 8.231, // start time in second
    mem: { rss: 55701504, heapTotal: 35713792, heapUsed: 19789608 }, // nodejs memory usage
    ops: [ 0, 0, 0 ], // operation per second in last 1, 5, 15 minutes
    tps: [ 0, 0, 0 ], // transaction per second in last 1, 5, 15 minutes
    lps: [ 0, 0, 0 ], // load(from backend) per second in last 1, 5, 15 minutes
    ups: [ 0, 0, 0 ], // unload(to backend) per second in last 1, 5, 15 minutes
    pps: [ 0, 0, 0 ], // persistent(to backend) per second in last 1, 5, 15 minutes
    counter: { // performance counter
        'find:player': [0, 0, 0], // 'method:collection' : [total time in ms, total count, average time in ms]
    }
}
memdb> db.resetCounter() //reset performance counter

memdb> ^D (to exit)

memdbcluster

Start memdb in cluster mode

When using memdbcluster, make sure you have ssh permission (without password) on all servers, and the memdb version, install folder, config files are all same in all servers

Usage

memdbcluster [command] [--conf=memdb.conf.js] [--shard=shardId]

Commands

  • start - Start cluster
  • stop - Stop cluster
  • status - Show cluster status
  • drop - Drop all data

Options

  • --conf - Memdb config file path
  • --shard(Optional) - Specify shardId. Will operate on all shards if not specified

Example

memdbcluster start

memdbcluster stop --shard=s1

memdbcluster status --conf=memdb.conf.js --shard=s1

memdbindex

Memdb index build tool. To update index on a non-empty database, or fix corrupted index data.

You must shutdown all shards before running memdbindex

Usage

memdbindex [rebuild | drop] [options]

Options

  • --conf - Memdb config file path
  • --coll - collection name
  • --keys - index keys, split with '.'

Example

memdbindex rebuild --conf=memdb.js --coll=player --keys=areaId
// The specified index on coll and keys must defined in memdb.js

memdbindex drop --coll=player --keys=areaId

memdb-client

Memdb nodejs client library

var memdb = require('memdb-client');

memdb.connect(opts) -> connection

Connect to memdb server

  • opts - Connect options
{ 
   host : '127.0.0.1',
   port : 31017,
}
  • Returns: connection

Example

var conn = yield memdb.connect({host : '127.0.0.1', port : 31017});

memdb.autoConnect(opts) -> autoConnection

Get autoConnection to memdb server

  • opts.shards - Specify all shards host and port
{
    shardId1 : {host : '127.0.0.1', port : 31017},
    shardId2 : {host : '127.0.0.1', port : 31018},
}
  • Returns: autoConnection

Example

var shards = {
    s1 : {host : '127.0.0.1', port : 31017},
    s2 : {host : '127.0.0.1', port : 31018},
};
var autoconn = yield memdb.autoConnect({shards : shards});

memdb.goose -> mdbgoose (sync)

Get mdbgoose instance

  • Returns: mdbgoose 
var mdbgoose = memdb.goose;

memdb.connectBackend(backend) -> mongodb

Connect directly to backend mongodb, return mongodb's db instance. You can use full power of mongodb's query system, please note that the data returned from backend is NOT guaranteed to be up-to-date, please adjust persistentDelay option to decrease data delay.

DO NOT write directly to backend mongodb, which can break data consistency.

  • backend.url - Mongodb connection string
  • Returns: Mongodb db instance

Example

var db = yield memdb.connectBackend({url : 'mongodb://localhost/test'});
var doc = yield db.collection('test').findOneAsync();

yield db.collection('test').removeAsync(); // Error! can not write

Connection

connection to memdb server

var conn = yield memdb.connect({host : '127.0.0.1', port : 31017});

conn.collection(name) -> collection (sync)

Get collection by name

  • name - Collection name
  • Returns: collection 
var collection = conn.collection('player');

conn.commit()

Commit changes

yield conn.commit();

conn.rollback()

Rollback changes

yield conn.rollback();

conn.close()

Close connection

yield conn.close();

AutoConnection

autoConnection to memdb server

var autoconn = yield memdb.autoConnect({
    shards : {
        s1 : {host : '127.0.0.1', port : 31017},
        s2 : {host : '127.0.0.1', port : 31018},
    }
});

autoconn.collection(name) -> collection (sync)

Get collection by name

  • name - collection name
var collection = yield autoconn.collection('player');

autoconn.transaction(func, shardId) -> result

Execute func in a transaction, auto commit on success or rollback on error.

The code inside func must be (domain)[https://nodejs.org/api/domain.html] safe.

  • func - The function to execute, it must return a promise
  • shardId - Specify shard to execute (required when you have multiple shards)
  • Returns: return whatever func returns

Generator style

try{
    var player = yield autoconn.transaction(Promise.coroutine(function*(){
        // Make queries here
        yield autoconn.collection('player').insert({_id : 1, name : 'rain'});
        return yield autoconn.collection('player').find(1);
    }), 's1');
}
catch(err){
    // Handle Error
}

Promise style

return autoconn.transaction(function(){
    return Promise.try(function(){
        return autoconn.collection('player').insert({_id : 1, name : 'rain'});
    })
    .then(function(){
        return autoconn.collection('player').find(1);
    });
}, 's1')
.then(function(ret){
    // ret is player
}, function(err){
    // Handle error
});

autoconn.close()

Close autoConnection

yield autoconn.close();

Collection

Collection of documents

var coll = conn.collection(name); //Sync API

coll.find(queryOrId[, fields][, opts]) -> document(s)

Find documents based on query.

  • queryOrId - Query or id. Only field-value pair is supported in query, others operations (like sorting, comparing) are not supported. If you specify a query, make sure the field combination used in the query is declared as index, query not using index is treated as error.
{_id : 1}; //OK!
{name : 'rain', age : 30}; //OK only if 'name' and 'age' is declared as compound index
{age : {$gte : 30}}; //Error! not supported
  • fields - fields to return
'field1 field2' // return field1 and field2
{key : true} // include key
{key : false} // exclude key
  • opts - Find options
  • opts.readonly - true/false (false by default). Specify readonly if you only read document (without modifying or do modification depend on the doc), this is faster. You can call coll.findReadOnly to do the same thing.
  • opts.limit - number. Limit number of returned documents.
{
    readonly : true,
    limit : 10,
}
  • Returns: document for id, [document] for query

Example:

var doc = yield coll.find(id);

var doc = yield coll.find(id, 'field');

var doc = yield coll.find(id, {field : false});

var doc = yield coll.find(id, null, {readonly : true});

var docs = yield coll.find({field : value});

coll.insert(docs) -> [doc._id]

Insert documents to collection

  • docs - doc or [doc]. A doc._id will be generated if you don't specify one.
  • Returns: [doc._id]

Example

yield coll.insert({_id : 1, name : 'rain'});

var ids = yield coll.insert([{name : 'rain'}, {name : 'snow'}]);

coll.update(queryOrId, modifier, opts) -> count

Update document

  • queryOrId - Refer to coll.find
  • modifier - Modifiers
  • doc - replace with the new doc
  • modifier.$set - set the specified fields
  • modifier.$unset - remove the specified fields
  • modifier.$inc - inc a field by value, create one if not exists
  • modifier.$push - push a item into array, create new array if not exists
  • modifier.$addToSet - push a item into array if not duplicate
  • modifier.$pop - pop a item from array
  • modifier.$pull - pull a item from array by value
yield coll.update(query, doc);

yield coll.update(query, {$set : {field : value}});

yield coll.update(query, {$unset : {field : true}});

yield coll.update(query, {$inc : {field : 5});

yield coll.update(query, {$push : {field : item}});

yield coll.update(query, {$addToSet : {field : item}});

yield coll.update(query, {$pop : {field : true}});

yield coll.update(query, {$pull : {field : value}});
  • opts - Options
  • opts.upsert - Insert if not exists, update if exists
yield coll.update(query, doc, {upsert : true});
  • Returns: count of updated

NOTE: DO NOT put large array in a document, which will slow down performance.

coll.remove(queryOrId) -> count

Remove documents

  • queryOrId - Refer to coll.find
  • Returns: count of removed
yield coll.remove(query);

coll.findOne(queryOrId[, fields][, opts]) -> document

Equals to (yield coll.find(queryOrId, fields, opts))[0];

coll.findById(id[, fields][, opts]) -> document

Equals to coll.find(id, fields, opts);

coll.findReadOnly(queryOrId[, fields][, opts]) -> document(s)

Equals to coll.find(queryOrId, fields, {readonly : true});

coll.findOneReadOnly(queryOrId[, fields][, opts]) -> document

Equals to (yield coll.find(queryOrId, fields, {readonly : true}))[0];

coll.findByIdReadOnly(id[, fields][, opts]) -> document

Equals to coll.find(id, fields, {readonly : true});

coll.count(query) -> number

Count documents based on query

  • query - The format of query equals to coll.find

Example:

var count = yield coll.find({field : value});

mdbgoose

The 'mongoose' for memdb.

Mdbgoose is modified from mongoose, so most mongoose API also work in mdbgoose, we just list the difference here. If you want to know how to work with mongoose, please refer to mongoose official doc

var mdbgoose = require('memdb').goose;

API Changes

Bluebird Promisified API

Mongoose doesn't support bluebird promise, so we wrap each API suffixed by 'Async' which return a bluebird promise, as below.

yield doc = Model.findAsync(id);
yield doc.saveAsync();

DO NOT use the old callback style

Model.find(id, function(err, ret){ 
    // Bad Practise
});

mdbgooose.connect(opts)

Specify the target servers, you must call connect before start any transaction. Please refer to memdb.autoConnect

  • opts.shards - Specify all shards host and port
{
    shardId1 : {host : '127.0.0.1', port : 31017},
    shardId2 : {host : '127.0.0.1', port : 31018},
}
  • opts.backend - The backend mongodb config (Optional)
{ url : 'mongodb://localhost/test' }

Example

var shards = {
    s1 : {host : '127.0.0.1', port : 31017},
    s2 : {host : '127.0.0.1', port : 31018},
};
var backend = {url : 'mongodb://localhost/test'}

yield mdbgoose.connectAsync({shards : shards, backend : backend});

mdbgoose.disconnect()

Disconnect from memdb server

yield mdbgoose.disconnectAsync();

mdbgoose.transaction(func[, shardId]) -> result

Execute func in a transaction, please refer to autoconn.transaction

yield mdbgoose.transactionAsync(P.coroutine(function*(){
    var doc = yield Model.findAsync(...);
    yield doc.saveAsync();
}), 's1');

mdbgoose.genCollectionConfig() -> collectionsConfig (sync)

Generate memdb collections config from mdbgoose schema. This is useful when you want to define indexes in schema.

Model.find

Please refer to coll.find

Model.findOne

Please refer to coll.findOne

Model.findById

Please refer to coll.findById

Model.findReadOnly

Please refer to coll.findReadOnly

Model.findOneReadOnly

Please refer to coll.findOneReadOnly

Model.findByIdReadOnly

Please refer to coll.findByIdReadOnly

Model.count

Please refer to coll.count

The following API is removed

  • findByIdAndRemove
  • findByIdAndUpdate
  • findOneAndUpdate
  • findOneAndRemove

Schema

Indexes

Index defined in model is for backend mongodb (If you like to make quries from backend mongodb, you can define proper indexes).

For memdb index, you should define in memdb.conf.js, note that live index change is NOT allowed.

Types

The following types are supported

  • String
  • Number
  • Array
  • Boolean
  • Mixed

The following types are NOT supported

  • Date
  • Buffer
  • ObjectId

Queries

MemDB support only field-value pair in query, and the queried fields must be declared as index. If you want more complex query, please refer to access backend mongodb

Model.find({_id : 1}); //OK!
Model.find({name : 'rain', age : 30}); //OK only if 'name' and 'age' is declared as compound index
Model.find({age : {$gte : 30}}); //Error! not supported

Access backend MongoDB

You can use mdbgoose to access backend MongoDB (instead of memdb), which is useful when you want complex and non-realtime queries. The data fetched directly from backend MongoDB is readonly, never write directly to backend MongoDB.

The API you can use is below, original mongoose API is renamed with 'Mongo' suffix

  • Model.find -> Model.findMongo
  • Model.findOne -> Model.findOneMongo
  • Model.findById -> Model.findByIdMongo
  • Model.count -> Model.countMongo

Restrictions

Memory Limit

Node has memory limition of about 1.4GB Each shard has heap limit of 1GB, which can load about 500k documents (larger docs can spend more memory), documents will flush back to backend storage when memory exceed this limit (this will slow down performance), please adjust idleTimeout parameter to balance memory usage and performance.

Collection name

Collection name can not contain with '$' or begin with 'system.'

field name

Field name can not start with '$' or contains '.'

request length

Each request message can not exceed 1MB after json encoded

document length

Document can not exceed 1MB after json encoded

Clone this wiki locally