Replication between ES 1.6 and Mongo 3.1.5 does not work

[stabenfeldt]

Hi,

I got MongoDB and ES running, started mongo-connector --auto-commit-interval=0 -m dockerhost:27017 -t dockerhost:9200 -d elastic_doc_manager. Any idea why I get this error message?

[ERROR] mongo_connector.oplog_manager:162 - OplogThread: Last entry no longer in oplog cannot recover! Collection(Database(MongoClient('dockerhost', 27017), u'local'), u'oplog.rs')
[ERROR] mongo_connector.connector:302 - MongoConnector: OplogThread <OplogThread(Thread-2, stopped 4564373504)> unexpectedly stopped! Shutting down
[ERROR] mongo_connector.oplog_manager:162 - OplogThread: Last entry no longer in oplog cannot recover! Collection(Database(MongoClient('dockerhost', 27017), u'local'), u'oplog.rs')
[ERROR] mongo_connector.connector:302 - MongoConnector: OplogThread <OplogThread(Thread-2, stopped 4580593664)> unexpectedly stopped! Shutting down
[ERROR] mongo_connector.oplog_manager:162 - OplogThread: Last entry no longer in oplog cannot recover! Collection(Database(MongoClient('dockerhost', 27017), u'local'), u'oplog.rs')
[ERROR] mongo_connector.connector:302 - MongoConnector: OplogThread <OplogThread(Thread-2, stopped 4380250112)> unexpectedly stopped! Shutting down

I have a docker-compose setup that starts MongoDB in a replicaset and ES that will make it easier to replicate. https://github.com/soldotno/elastic-mongo/tree/mongo-connector

This is what I did:

git clone https://github.com/soldotno/elastic-mongo.git
cd elastic-mongo
git checkout mongo-connector
docker-compose up -d 

docker ps  # =>
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS                  PORTS                                                NAMES
39ed9e7c6f33        elasticsearch:1.6   "/docker-entrypoint.   1 seconds ago       Up Less than a second   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp       elasticmongo_elasticsearch_1
48f2b777abdf        mongo:3.1.5         "/usr/bin/mongod --r   2 seconds ago       Up 1 seconds            0.0.0.0:27017->27017/tcp, 0.0.0.0:28017->28017/tcp   elasticmongo_mongo1_1
7e6dd54b85e5        mongo:3.1.5         "/usr/bin/mongod --r   2 seconds ago       Up 1 seconds            0.0.0.0:27018->27017/tcp, 0.0.0.0:28018->28017/tcp   elasticmongo_mongo3_1
24e147982c2d        mongo:3.1.5         "/usr/bin/mongod --r   3 seconds ago       Up 2 seconds            0.0.0.0:27019->27017/tcp, 0.0.0.0:28019->28017/tcp   elasticmongo_mongo2_1

Start the mongo-connector

 mongo-connector --auto-commit-interval=0 -m dockerhost:27017 -t dockerhost:9200 -d elastic_doc_manager

[stabenfeldt]

The first attempt was done by running mono-connector from my workstation.
Next attempt was executed from mongo1 (the primary database).

Find the IP for the ES server in the cluster. Run this from the same machine as you executed docker-compose up -d

docker inspect elasticmongo_elasticsearch_1 | ack IPA | head -1  # => "IPAddress": "172.17.0.66",

Start a shell on the container running mongo as primary.

 docker   exec -i -t elasticmongo_mongo1_1  bash

mongo-connector  --auto-commit-interval=0   -m localhost:27017 -t 172.17.0.66:9200 -d elastic_doc_manager

This gives me no output in the log what so ever. Am I supposed to run monog-connector from the primary MongoDB, or can it be running elsewhere?

I've tried to insert data into different databases at the primary mono-database while the transporter is running. Nothing happens.

[llvtt]

Try running mongo-connector with the verbose flag (-v). This should provide
more output in mongo-connector.log.

On Wed, Jul 15, 2015 at 5:30 AM, Martin Stabenfeldt <
[email protected]> wrote:

The first attempt was done by running mono-connector from my workstation.
Next attempt was executed from mongo1 (the primary database). That gave me
no output in the log what so ever.

—
Reply to this email directly or view it on GitHub
#287 (comment)
.

[stabenfeldt]

Hi,

Nope, that didn't give any output either. :-(

[llvtt]

Can you try adding the --stdout option? This will send output directly to
stdout, rather than a log file, in case there were any problems writing to
the log file.

Does mongo-connector exit immediately after running the command, or does it
appear to be running, i.e., the command blocks? Also check that you are
able to access all shards and the mongos, or your single replica set, from
a "mongo" shell.

On Wed, Jul 15, 2015 at 10:26 AM, Martin Stabenfeldt <
[email protected]> wrote:

Hi,

Nope, that didn't give any output. :-(

—
Reply to this email directly or view it on GitHub
#287 (comment)
.

[stabenfeldt]

Here´s the output:

2015-07-15 17:58:38,041 [DEBUG] mongo_connector.oplog_manager:155 - OplogThread: Getting cursor
2015-07-15 17:58:38,041 [DEBUG] mongo_connector.oplog_manager:661 - OplogThread: reading last checkpoint as Timestamp(1436962940, 1)
2015-07-15 17:58:38,042 [DEBUG] mongo_connector.oplog_manager:647 - OplogThread: oplog checkpoint updated to Timestamp(1436962940, 1)
2015-07-15 17:58:38,044 [DEBUG] mongo_connector.oplog_manager:167 - OplogThread: Last entry is the one we already processed.  Up to date.  Sleeping.
2015-07-15 17:58:39,046 [DEBUG] mongo_connector.oplog_manager:155 - OplogThread: Getting cursor
2015-07-15 17:58:39,046 [DEBUG] mongo_connector.oplog_manager:661 - OplogThread: reading last checkpoint as Timestamp(1436962940, 1)
2015-07-15 17:58:39,046 [DEBUG] mongo_connector.oplog_manager:647 - OplogThread: oplog checkpoint updated to Timestamp(1436962940, 1)
2015-07-15 17:58:39,051 [DEBUG] mongo_connector.oplog_manager:167 - OplogThread: Last entry is the one we already processed.  Up to date.  Sleeping.

mongo-connector keeps running and writes to the screen continuously.
Yes, I'm also able to mongo dockerhost and both write and read data. :)
dockerhost:27017 is the primary one.

dockerhost is the IP to my virtual machine (boot2docker) running the docker containers.

[stabenfeldt]

Connecting to dockerhost:27018 and dockerhost:27019 gives me this prompt:
rs:SECONDARY>

dockerhost:27017 => rs:PRIMARY>

[stabenfeldt]

Can you give it a go with the docker approach I described? Shouldn't take many minutes when you have docker and docker-compse installed. :-)

[stabenfeldt]

This is strange. It seams to work with the following command:
mongo-connector --auto-commit-interval=0 -m localhost:27017 -t 172.17.0.66:9200 -d elastic_doc_manager -vv --stdout Dunno why it didn't work previously.

I'll continue the work on getting ES / MongoDB be friends tomorrow. Thanks for helping out! 👍