failed to create river [mongodb][mongodb] #126

fmpwizard · 2013-09-17T18:34:27Z

Hi,

I'm getting this error when trying to setup the plugin to sync up mongo and elastic search.

OS Fedora 19 64 bit
elastic ver: 0.90.4
plugin: master from today, last commit:

commit 90554df863f3657f4ca3a729a98291eaa565c782
Author: Richard Louapre <[email protected]>
Date:   Mon Sep 16 10:19:49 2013 -0400

    Move ts as first expression in oplog.rs query

mongo replica is running:

rs0:PRIMARY> rs.status
function () { return db._adminCommand("replSetGetStatus"); }
rs0:PRIMARY> rs.initialize()
Tue Sep 17 14:27:59.154 TypeError: Object function () { return "try rs.help()"; } has no method 'initialize'
rs0:PRIMARY> rs.initiate()
{
    "info" : "try querying local.system.replset to see current configuration",
    "ok" : 0,
    "errmsg" : "already initialized"
}
rs0:PRIMARY> rs.status
function () { return db._adminCommand("replSetGetStatus"); }
rs0:PRIMARY> rs.conf()
{
    "_id" : "rs0",
    "version" : 1,
    "members" : [
        {
            "_id" : 0,
            "host" : "localhost.localdomain:27017"
        }
    ]
}
rs0:PRIMARY>

The elastic log is:

[2013-09-17 14:26:00,953][DEBUG][action.index             ] [Toro] Sending mapping updated to master: index [_river] type [mongodb]
[2013-09-17 14:26:00,961][INFO ][cluster.metadata         ] [Toro] [_river] update_mapping [mongodb] (dynamic)
[2013-09-17 14:26:01,005][DEBUG][action.index             ] [Toro] Sending mapping updated to master: index [_river] type [mongodb]
[2013-09-17 14:26:01,006][INFO ][river.mongodb            ] [Toro] [mongodb][mongodb] Using mongodb server(s): host [localhost], port [27017]
[2013-09-17 14:26:01,007][INFO ][river.mongodb            ] [Toro] [mongodb][mongodb] MongoDB River Plugin version: [1.7.1-SNAPSHOT]
[2013-09-17 14:26:01,007][INFO ][river.mongodb            ] [Toro] [mongodb][mongodb] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [false], include_collection [], throttlesize [500], gridfs [false], filter [], db [acm-web], collection [as.alerts], script [null], indexing to [as]/[alerts]
[2013-09-17 14:26:01,037][INFO ][cluster.metadata         ] [Toro] [_river] update_mapping [mongodb] (dynamic)
[2013-09-17 14:26:01,056][WARN ][river                    ] [Toro] failed to create river [mongodb][mongodb]
java.lang.NullPointerException
    at com.mongodb.MongoOptions.<init>(MongoOptions.java:42)
    at com.mongodb.MongoClient.<init>(MongoClient.java:253)
    at com.mongodb.MongoClient.<init>(MongoClient.java:230)
    at org.elasticsearch.river.mongodb.MongoDBRiver.getMongoClient(MongoDBRiver.java:357)
    at org.elasticsearch.river.mongodb.MongoDBRiver.getAdminDb(MongoDBRiver.java:312)
    at org.elasticsearch.river.mongodb.MongoDBRiver.isMongos(MongoDBRiver.java:290)
    at org.elasticsearch.river.mongodb.MongoDBRiver.start(MongoDBRiver.java:250)
    at org.elasticsearch.river.RiversService.createRiver(RiversService.java:146)
    at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:270)
    at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:264)
    at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:93)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
[2013-09-17 14:26:01,063][DEBUG][action.index             ] [Toro] Sending mapping updated to master: index [_river] type [mongodb]
[2013-09-17 14:26:01,096][INFO ][cluster.metadata         ] [Toro] [_river] update_mapping [mongodb] (dynamic)

when I run:

curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{
    "type": "mongodb", 
    "mongodb": { 
        "db": "acm-web", 
        "collection": "as.alerts", 
        "gridfs": false
    }, 
    "index": { 
        "name": "as", 
        "type": "alerts" 
    }
}'
{"ok":true,"_index":"_river","_type":"mongodb","_id":"_meta","_version":2}

so, I get an ok, but the error shows up on the logs for elastic, and restarting elastic gives me

[2013-09-17 14:33:52,001][DEBUG][action.get               ] [Phineas T. Horton] [_river][0]: failed to execute [[_river][mongodb][_meta]: routing [null]]
org.elasticsearch.indices.IndexMissingException: [_river] missing
    at org.elasticsearch.indices.InternalIndicesService.indexServiceSafe(InternalIndicesService.java:334)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:96)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:42)
    at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:161)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
[2013-09-17 14:33:52,423][DEBUG][action.get               ] [Phineas T. Horton] [_river][0]: failed to execute [[_river][mongodb][_meta]: routing [null]]
org.elasticsearch.index.shard.IllegalIndexShardStateException: [_river][0] CurrentState[RECOVERING] operations only allowed when started/relocated
    at org.elasticsearch.index.shard.service.InternalIndexShard.readAllowed(InternalIndexShard.java:704)
    at org.elasticsearch.index.shard.service.InternalIndexShard.get(InternalIndexShard.java:415)
    at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:167)
    at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:101)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:103)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:42)
    at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:161)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
[2013-09-17 14:33:52,424][WARN ][river.routing            ] [Phineas T. Horton] failed to get/parse _meta for [mongodb]
org.elasticsearch.index.shard.IllegalIndexShardStateException: [_river][0] CurrentState[RECOVERING] operations only allowed when started/relocated
    at org.elasticsearch.index.shard.service.InternalIndexShard.readAllowed(InternalIndexShard.java:704)
    at org.elasticsearch.index.shard.service.InternalIndexShard.get(InternalIndexShard.java:415)
    at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:167)
    at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:101)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:103)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:42)
    at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:161)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
[2013-09-17 14:33:52,690][INFO ][river.mongodb            ] [Phineas T. Horton] [mongodb][mongodb] Using mongodb server(s): host [localhost], port [27017]
[2013-09-17 14:33:52,691][INFO ][river.mongodb            ] [Phineas T. Horton] [mongodb][mongodb] MongoDB River Plugin version: [1.7.1-SNAPSHOT]
[2013-09-17 14:33:52,691][INFO ][river.mongodb            ] [Phineas T. Horton] [mongodb][mongodb] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [false], include_collection [], throttlesize [500], gridfs [false], filter [], db [acm-web], collection [as.alerts], script [null], indexing to [as]/[alerts]
[2013-09-17 14:33:52,728][WARN ][river                    ] [Phineas T. Horton] failed to create river [mongodb][mongodb]
java.lang.NullPointerException
    at com.mongodb.MongoOptions.<init>(MongoOptions.java:42)
    at com.mongodb.MongoClient.<init>(MongoClient.java:253)
    at com.mongodb.MongoClient.<init>(MongoClient.java:230)
    at org.elasticsearch.river.mongodb.MongoDBRiver.getMongoClient(MongoDBRiver.java:357)
    at org.elasticsearch.river.mongodb.MongoDBRiver.getAdminDb(MongoDBRiver.java:312)
    at org.elasticsearch.river.mongodb.MongoDBRiver.isMongos(MongoDBRiver.java:290)
    at org.elasticsearch.river.mongodb.MongoDBRiver.start(MongoDBRiver.java:250)
    at org.elasticsearch.river.RiversService.createRiver(RiversService.java:146)
    at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:270)
    at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:264)
    at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:93)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)

Any help would be great, thanks

The text was updated successfully, but these errors were encountered:

fmpwizard · 2013-09-17T18:39:35Z

More info, the pligins folder:

 ll plugins/mongo/
total 896
-rw-rw-r--. 1 diego diego  51826 Sep 17 09:40 elasticsearch-river-mongodb-1.7.1-SNAPSHOT.jar
-rw-rw-r--. 1 diego diego 437897 Sep 17 09:42 elasticsearch-river-mongodb-1.7.1-SNAPSHOT.zip
-rw-rw-r--. 1 diego diego 419054 Sep 17 09:39 mongo-java-driver-2.11.3.jar
drwxr-xr-x. 4 diego diego   4096 Sep 17 09:38 _site

Mongodb version 2.4.6

richardwilly98 · 2013-09-17T19:04:08Z

Can you please try to add the options section in the river settings?

I believe there is a bug due to one of the last refactoring committed.

Btw that should work with the last version released.

Sent from my iPad

On Sep 17, 2013, at 2:39 PM, Diego Medina [email protected] wrote:

More info, the pligins folder:

ll plugins/mongo/
total 896
-rw-rw-r--. 1 diego diego 51826 Sep 17 09:40 elasticsearch-river-mongodb-1.7.1-SNAPSHOT.jar
-rw-rw-r--. 1 diego diego 437897 Sep 17 09:42 elasticsearch-river-mongodb-1.7.1-SNAPSHOT.zip
-rw-rw-r--. 1 diego diego 419054 Sep 17 09:39 mongo-java-driver-2.11.3.jar
drwxr-xr-x. 4 diego diego 4096 Sep 17 09:38 _site

Mongodb version 2.4.6

—
Reply to this email directly or view it on GitHub.

fmpwizard · 2013-09-18T02:18:09Z

Hi,

Thanks for the quick reply, I downgraded to 1.8.6 and even though I get the error:

[2013-09-17 21:32:40,071][DEBUG][action.get               ] [Scarlotti, Mark] [_river][0]: failed to execute [[_river][mongodb][_meta]: routing [null]]
org.elasticsearch.indices.IndexMissingException: [_river] missing
    at org.elasticsearch.indices.InternalIndicesService.indexServiceSafe(InternalIndicesService.java:334)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:96)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:42)
    at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:161)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)

a few lines later elastic starts indexing the mongodb documents.

I looked at the docs for the "options" but I wasn't sure what to put in there as values for each entry. IF you could give me some sample of what to put, I can try the latest again tomorrow and give you more information.

And I also have an unrelated question (not sure where to ask, so here it goes), how do you do a full mongo index into elastic)? Let's say I have my mongo db with 1k documents, then I installed the plugin and elastic, but elastic will only get documents I edit/create from "now on".

Thanks you very much!

DIego

- NPE if options is missing in river settings.

richardwilly98 · 2013-09-18T17:38:08Z

Hi,

With release 1.7.0 - please try:

curl -XDELETE 'http://localhost:9200/_river/mongodb

curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{
    "type": "mongodb", 
    "mongodb": { 
        "options": {}, 
        "db": "acm-web", 
        "collection": "as.alerts", 
        "gridfs": false
    }, 
    "index": { 
        "name": "as", 
        "type": "alerts" 
    }
}'

Can you please provide the full ES log file if it failed again?

richardwilly98 · 2013-09-28T12:36:37Z

@fmpwizard any update with the suggested river settings?

fmpwizard · 2013-09-28T12:41:56Z

Sorry I haven't been able to try it out, but will post an update as soon as I can put some time aside.

fmpwizard · 2013-10-02T01:08:36Z

@richardwilly98 Hi, 1.7.0 works great! there are some exceptions in the elastic logs, but the river works fine and all items are being updated.

And about the initial sync, what I did was:

1- enable replica on mongo
2- setup the river /restart, etc
3- restore mongo from a backup
4- all documents were sent to elastic

Thanks!

richardwilly98 added a commit that referenced this issue Sep 18, 2013

Fix for #126

f9d782f

- NPE if options is missing in river settings.

richardwilly98 closed this as completed Oct 7, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

failed to create river [mongodb][mongodb] #126

failed to create river [mongodb][mongodb] #126

fmpwizard commented Sep 17, 2013

fmpwizard commented Sep 17, 2013

richardwilly98 commented Sep 17, 2013

fmpwizard commented Sep 18, 2013

richardwilly98 commented Sep 18, 2013

richardwilly98 commented Sep 28, 2013

fmpwizard commented Sep 28, 2013

fmpwizard commented Oct 2, 2013

failed to create river [mongodb][mongodb] #126

failed to create river [mongodb][mongodb] #126

Comments

fmpwizard commented Sep 17, 2013

fmpwizard commented Sep 17, 2013

richardwilly98 commented Sep 17, 2013

fmpwizard commented Sep 18, 2013

richardwilly98 commented Sep 18, 2013

richardwilly98 commented Sep 28, 2013

fmpwizard commented Sep 28, 2013

fmpwizard commented Oct 2, 2013