No documents found on elastic #111

arnaudroubinet · 2013-08-07T15:54:53Z

Hello Richard,

I use elasticsearch-0.90.2 with
elasticsearch-river-mongodb-1.6.11
mongo-java-driver-2.11.2
And elasticsearch-mapper-attachments-1.7.0.

I have two replicates of my mongodb(2.4.5) on the same computer and one instance of elasticsearch.

When I send my json files elastic say ok, I have on log the server status and my mapping is updated, but after that, nothing happen.
I check replication between my two instances of mongo and it work fine.
I check oplogs and seem work fine.

I have no error on the log and I’m very disappointed.

Can you help me?

{
  "type": "mongodb",
  "mongodb": {
    "servers": [
      {
        "host": "127.0.0.1",
        "port": "27017"
      },
      {
        "host": "127.0.0.1",
        "port": "27018"
      }
    ],
    "options": {
      "secondary_read_preference": true
    },
    "credentials": [
      {
        "db": "b2c_dev",
        "user": "b2c_dev",
        "password": "B2C_Dev"
      }
    ],
    "db": "b2c_dev",
    "collection": "shopping_users",
    "gridfs": false
  },
  "index": {
    "name": "personindex"
  }
}

richardwilly98 · 2013-08-07T16:03:40Z

@arnaudroubinet could you please provide ES log file?

You also have authentication enabled in MongoDB. Can you confirm the user can access oplog.rs ?

Thanks,
Richard.

arnaudroubinet · 2013-08-07T16:13:10Z

This is my log.
I check access for oplog.rs

[2013-08-07 17:12:29,129][INFO ][node [2013-08-07 17:12:29,223][INFO ][plugins [2013-08-07 17:12:32,970][INFO ][node [2013-08-07 17:12:32,971][INFO ][node [2013-08-07 17:12:33,252][INFO ][transport [2013-08-07 17:12:36,319][INFO ][cluster.service [2013-08-07 17:12:36,373][INFO ][discovery [2013-08-07 17:12:36,399][INFO ][http [2013-08-07 17:12:36,400][INFO ][node [2013-08-07 17:12:36,480][INFO ][gateway [2013-08-07 17:14:29,790][INFO ][cluster.metadata [2013-08-07 17:14:30,407][INFO ][cluster.metadata [2013-08-07 17:14:30,455][DEBUG][river.mongodb [2013-08-07 17:14:30,455][DEBUG][river.mongodb [2013-08-07 17:14:30,458][INFO ][river.mongodb [2013-08-07 17:14:30,458][INFO ][river.mongodb [2013-08-07 17:14:30,571][INFO ][cluster.metadata [2013-08-07 17:14:30,752][DEBUG][river.mongodb [2013-08-07 17:14:30,753][INFO ][river.mongodb [2013-08-07 17:14:30,785][INFO ][cluster.metadata [2013-08-07 17:14:31,109][TRACE][river.mongodb [2013-08-07 17:14:31,117][TRACE][river.mongodb [2013-08-07 17:14:31,212][INFO ][river.mongodb [2013-08-07 17:14:31,218][DEBUG][river.mongodb [2013-08-07 17:14:31,282][INFO ][cluster.metadata ] [Blaze, Johnny] {0.90.2}[26563]: initializing ...
] [Blaze, Johnny] loaded [river-mongodb, mapper-attachments], sites [head]
] [Blaze, Johnny] {0.90.2}[26563]: initialized
] [Blaze, Johnny] {0.90.2}[26563]: starting ...
] [Blaze, Johnny] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.200.123:9300]}
] [Blaze, Johnny] new_master [Blaze, Johnny][IMHRMLidSmyjwD8UmZ1OnA][inet[/192.168.200.123:9300]], reason: zen-disco-join (elected_as_master)
] [Blaze, Johnny] elasticsearch/IMHRMLidSmyjwD8UmZ1OnA
] [Blaze, Johnny] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.200.123:9200]}
] [Blaze, Johnny] {0.90.2}[26563]: started
] [Blaze, Johnny] recovered [0] indices into cluster_state
] [Blaze, Johnny] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings []
] [Blaze, Johnny] [_river] update_mapping mongogridfs
] [Blaze, Johnny] [mongodb][mongogridfs] Prefix: [[Blaze, Johnny] [mongodb][mongogridfs] ] - name: [river.mongodb]
] River settings:
] [Blaze, Johnny] [mongodb][mongogridfs] Using mongodb server(s): host [localhost], port [27017]
] [Blaze, Johnny] [mongodb][mongogridfs] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [false], throttlesize [500], gridfs [true], filter [], db [testmongo], collection [files], script [null], indexing to [testmongo]/[files]
] [Blaze, Johnny] [testmongo] creating index, cause [api], shards [5]/[1], mappings []
] [Blaze, Johnny] [mongodb][mongogridfs] Set explicit attachment mapping.
] [Blaze, Johnny] [mongodb][mongogridfs] Mapping: {"files":{"properties":{"content":{"type":"attachment"},"filename":{"type":"string"},"contentType":{"type":"string"},"md5":{"type":"string"},"length":{"type":"long"},"chunkSize":{"type":"long"}}}}
] [Blaze, Johnny] [testmongo] create_mapping [files]
] [Blaze, Johnny] [mongodb][mongogridfs] serverStatus: { "serverUsed" : "esearch/127.0.1.1:27017" , "host" : "esearch" , "version" : "2.4.5" , "process" : "mongod" , "pid" : 23908 , "uptime" : 5003.0 , "uptimeMillis" : 5002396 , "uptimeEstimate" : 4948.0 , "localTime" : { "$date" : "2013-08-07T15:14:31.077Z"} , "asserts" : { "regular" : 0 , "warning" : 0 , "msg" : 0 , "user" : 111 , "rollovers" : 0} , "backgroundFlushing" : { "flushes" : 83 , "total_ms" : 13 , "average_ms" : 0.1566265060240964 , "last_ms" : 0 , "last_finished" : { "$date" : "2013-08-07T15:14:08.709Z"}} , "connections" : { "current" : 8 , "available" : 15992 , "totalCreated" : 302} , "cursors" : { "totalOpen" : 3 , "clientCursors_size" : 3 , "timedOut" : 8} , "dur" : { "commits" : 30 , "journaledMB" : 0.0 , "writeToDataFilesMB" : 0.0 , "compression" : 0.0 , "commitsInWriteLock" : 0 , "earlyCommits" : 0 , "timeMs" : { "dt" : 3068 , "prepLogBuffer" : 0 , "writeToJournal" : 0 , "writeToDataFiles" : 0 , "remapPrivateView" : 0}} , "extra_info" : { "note" : "fields vary by platform" , "heap_usage_bytes" : 65331496 , "page_faults" : 5} , "globalLock" : { "totalTime" : 5002396000 , "lockTime" : 178767 , "currentQueue" : { "total" : 0 , "readers" : 0 , "writers" : 0} , "activeClients" : { "total" : 0 , "readers" : 0 , "writers" : 0}} , "indexCounters" : { "accesses" : 12 , "hits" : 12 , "misses" : 0 , "resets" : 0 , "missRatio" : 0.0} , "locks" : { "." : { "timeLockedMicros" : { "R" : 123357 , "W" : 178767} , "timeAcquiringMicros" : { "R" : 66872 , "W" : 15566}} , "admin" : { "timeLockedMicros" : { } , "timeAcquiringMicros" : { }} , "local" : { "timeLockedMicros" : { "r" : 241224 , "w" : 5529} , "timeAcquiringMicros" : { "r" : 38001 , "w" : 276}} , "dev_b2c" : { "timeLockedMicros" : { "r" : 14182 , "w" : 1742} , "timeAcquiringMicros" : { "r" : 328 , "w" : 36}} , "test" : { "timeLockedMicros" : { "r" : 1430 , "w" : 0} , "timeAcquiringMicros" : { "r" : 68 , "w" : 0}}} , "network" : { "bytesIn" : 2414516 , "bytesOut" : 9007054 , "numRequests" : 39024} , "opcounters" : { "insert" : 1 , "query" : 221 , "update" : 15 , "delete" : 0 , "getmore" : 1189 , "command" : 37778} , "opcountersRepl" : { "insert" : 0 , "query" : 0 , "update" : 0 , "delete" : 0 , "getmore" : 0 , "command" : 0} , "recordStats" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0 , "dev_b2c" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "local" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "test" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0}} , "repl" : { "setName" : "rs0" , "ismaster" : true , "secondary" : false , "hosts" : [ "esearch:27017" , "esearch:27018"] , "primary" : "esearch:27017" , "me" : "esearch:27017"} , "writeBacksQueued" : false , "mem" : { "bits" : 64 , "resident" : 42 , "virtual" : 5122 , "supported" : true , "mapped" : 2287 , "mappedWithJournal" : 4574} , "metrics" : { "document" : { "deleted" : 0 , "inserted" : 1 , "returned" : 88 , "updated" : 15} , "getLastError" : { "wtime" : { "num" : 0 , "totalMillis" : 0} , "wtimeouts" : 0} , "operation" : { "fastmod" : 10 , "idhack" : 12 , "scanAndOrder" : 5} , "queryExecutor" : { "scanned" : 1232} , "record" : { "moves" : 1} , "repl" : { "apply" : { "batches" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0} , "buffer" : { "count" : 0 , "maxSizeBytes" : 268435456 , "sizeBytes" : 0} , "network" : { "bytes" : 0 , "getmores" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0 , "readersCreated" : 27} , "oplog" : { "insert" : { "num" : 9 , "totalMillis" : 0} , "insertBytes" : 2322} , "preload" : { "docs" : { "num" : 0 , "totalMillis" : 0} , "indexes" : { "num" : 0 , "totalMillis" : 0}}} , "ttl" : { "deletedDocuments" : 0 , "passes" : 83}} , "ok" : 1.0}
] [Blaze, Johnny] [mongodb][mongogridfs] process: mongod
] [Blaze, Johnny] [mongodb][mongogridfs] No known previous slurping time for this collection
] [Blaze, Johnny] [mongodb][mongogridfs] Using filter: { "$and" : [ { "ns" : "testmongo.files.files"}]}
] [Blaze, Johnny] [_river] update_mapping mongogridfs

richardwilly98 · 2013-08-07T16:25:34Z

@arnaudroubinet the log does not match the river settings you have provided. The log says:

gridfs: true
db: testmongo
collection: files

arnaudroubinet · 2013-08-07T16:36:36Z

Thank you for helping me.

I can't login on mongooplog and credential I provided before.

But when I use mongooplog --from localhost I have this return.

connected to: 127.0.0.1
Wed Aug 7 18:32:50.343 [oplogreplay] going to connect
Wed Aug 7 18:32:50.344 [oplogreplay] connected
Wed Aug 7 18:32:50.344 [oplogreplay] starting from Aug 6 18:32:50:0

This time I try without credential and I have this result.

[2013-08-07 18:30:00,874][INFO ][cluster.metadata ] [Blaze, Johnny] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings []
[2013-08-07 18:30:00,980][INFO ][cluster.metadata ] [Blaze, Johnny] [_river] update_mapping mongodb
[2013-08-07 18:30:00,988][DEBUG][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] Prefix: [[Blaze, Johnny] [mongodb][mongodb] ] - name: [river.mongodb]
[2013-08-07 18:30:00,988][DEBUG][river.mongodb ] River settings:
[2013-08-07 18:30:00,988][INFO ][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] mongoServersSettings: [{port=27017, host=127.0.0.1}, {port=27018, host=127.0.0.1}]
[2013-08-07 18:30:00,988][INFO ][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] Server: 127.0.0.1 - 27017
[2013-08-07 18:30:00,989][INFO ][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] Server: 127.0.0.1 - 27018
[2013-08-07 18:30:00,989][INFO ][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] Using mongodb server(s): host [127.0.0.1], port [27017]
[2013-08-07 18:30:00,990][INFO ][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] Using mongodb server(s): host [127.0.0.1], port [27018]
[2013-08-07 18:30:00,991][INFO ][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] starting mongodb stream. options: secondaryreadpreference [true], drop_collection [false], throttlesize [500], gridfs [false], filter [], db [b2c_dev], collection [shopping_users], script [null], indexing to [personindex]/[b2c_dev]
[2013-08-07 18:30:01,091][INFO ][cluster.metadata ] [Blaze, Johnny] [personindex] creating index, cause [api], shards [5]/[1], mappings []
[2013-08-07 18:30:01,327][TRACE][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] serverStatus: { "serverUsed" : "esearch/127.0.1.1:27017" , "host" : "esearch" , "version" : "2.4.5" , "process" : "mongod" , "pid" : 23908 , "uptime" : 9533.0 , "uptimeMillis" : 9532638 , "uptimeEstimate" : 9430.0 , "localTime" : { "$date" : "2013-08-07T16:30:01.319Z"} , "asserts" : { "regular" : 0 , "warning" : 0 , "msg" : 0 , "user" : 111 , "rollovers" : 0} , "backgroundFlushing" : { "flushes" : 158 , "total_ms" : 32 , "average_ms" : 0.20253164556962025 , "last_ms" : 0 , "last_finished" : { "$date" : "2013-08-07T16:29:08.726Z"}} , "connections" : { "current" : 11 , "available" : 15989 , "totalCreated" : 480} , "cursors" : { "totalOpen" : 3 , "clientCursors_size" : 3 , "timedOut" : 10} , "dur" : { "commits" : 30 , "journaledMB" : 0.0 , "writeToDataFilesMB" : 0.0 , "compression" : 0.0 , "commitsInWriteLock" : 0 , "earlyCommits" : 0 , "timeMs" : { "dt" : 3068 , "prepLogBuffer" : 0 , "writeToJournal" : 0 , "writeToDataFiles" : 0 , "remapPrivateView" : 0}} , "extra_info" : { "note" : "fields vary by platform" , "heap_usage_bytes" : 65419736 , "page_faults" : 13} , "globalLock" : { "totalTime" : 9532638000 , "lockTime" : 443100 , "currentQueue" : { "total" : 0 , "readers" : 0 , "writers" : 0} , "activeClients" : { "total" : 0 , "readers" : 0 , "writers" : 0}} , "indexCounters" : { "accesses" : 925 , "hits" : 925 , "misses" : 0 , "resets" : 0 , "missRatio" : 0.0} , "locks" : { "." : { "timeLockedMicros" : { "R" : 238860 , "W" : 443100} , "timeAcquiringMicros" : { "R" : 125663 , "W" : 64998}} , "admin" : { "timeLockedMicros" : { "r" : 791 , "w" : 0} , "timeAcquiringMicros" : { "r" : 9 , "w" : 0}} , "local" : { "timeLockedMicros" : { "r" : 655108 , "w" : 8113} , "timeAcquiringMicros" : { "r" : 181992 , "w" : 373}} , "testmongo" : { "timeLockedMicros" : { "r" : 6559 , "w" : 10560} , "timeAcquiringMicros" : { "r" : 452 , "w" : 26}} , "dev_b2c" : { "timeLockedMicros" : { "r" : 27734 , "w" : 1742} , "timeAcquiringMicros" : { "r" : 711 , "w" : 36}} , "test" : { "timeLockedMicros" : { "r" : 10021 , "w" : 0} , "timeAcquiringMicros" : { "r" : 437 , "w" : 0}}} , "network" : { "bytesIn" : 5683232 , "bytesOut" : 15824681 , "numRequests" : 46565} , "opcounters" : { "insert" : 220 , "query" : 569 , "update" : 801 , "delete" : 0 , "getmore" : 3364 , "command" : 44424} , "opcountersRepl" : { "insert" : 0 , "query" : 0 , "update" : 0 , "delete" : 0 , "getmore" : 0 , "command" : 0} , "recordStats" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0 , "admin" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "dev_b2c" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "local" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "test" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "testmongo" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0}} , "repl" : { "setName" : "rs0" , "ismaster" : true , "secondary" : false , "hosts" : [ "esearch:27017" , "esearch:27018"] , "primary" : "esearch:27017" , "me" : "esearch:27017"} , "writeBacksQueued" : false , "mem" : { "bits" : 64 , "resident" : 51 , "virtual" : 5446 , "supported" : true , "mapped" : 2447 , "mappedWithJournal" : 4894} , "metrics" : { "document" : { "deleted" : 0 , "inserted" : 7 , "returned" : 3470 , "updated" : 21} , "getLastError" : { "wtime" : { "num" : 0 , "totalMillis" : 0} , "wtimeouts" : 0} , "operation" : { "fastmod" : 16 , "idhack" : 12 , "scanAndOrder" : 5} , "queryExecutor" : { "scanned" : 3810} , "record" : { "moves" : 1} , "repl" : { "apply" : { "batches" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0} , "buffer" : { "count" : 0 , "maxSizeBytes" : 268435456 , "sizeBytes" : 0} , "network" : { "bytes" : 0 , "getmores" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0 , "readersCreated" : 27} , "oplog" : { "insert" : { "num" : 1008 , "totalMillis" : 0} , "insertBytes" : 2797686} , "preload" : { "docs" : { "num" : 0 , "totalMillis" : 0} , "indexes" : { "num" : 0 , "totalMillis" : 0}}} , "ttl" : { "deletedDocuments" : 0 , "passes" : 158}} , "ok" : 1.0}
[2013-08-07 18:30:01,327][TRACE][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] process: mongod
[2013-08-07 18:30:01,373][INFO ][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] No known previous slurping time for this collection
[2013-08-07 18:30:01,374][DEBUG][river.mongodb ] [Blaze, Johnny] [mongodb][mongodb] Using filter: { "$and" : [ { "$or" : [ { "ns" : "b2c_dev.shopping_users"} , { "ns" : "b2c_dev.$cmd"}]}]}
[2013-08-07 18:30:01,498][INFO ][cluster.metadata ] [Blaze, Johnny] [_river] update_mapping mongodb

For this json files

{
"type": "mongodb",
"mongodb": {
"servers": [
{
"host": "127.0.0.1",
"port": "27017"
},
{
"host": "127.0.0.1",
"port": "27018"
}
],
"options": {
"secondary_read_preference": true
},
"db": "b2c_dev",
"collection": "shopping_users",
"gridfs": false
},
"index": {
"name": "personindex"
}
}

richardwilly98 · 2013-08-07T16:51:23Z

From the log we can tell the river is polling documents from b2c_dev.shopping_users.
Do you have documents stored in MongoDB (database/collection) b2c_dev.shopping_users?

arnaudroubinet · 2013-08-07T17:27:24Z

I can't check the good syntax of name before tomorrow, but yes, I have ten documents in this collection.

richardwilly98 · 2013-08-07T18:31:53Z

Please also execute the following query from MongoDB local database:

db.oplog.rs.find({"ns": "b2c_dev.shopping_users"}).pretty()

arnaudroubinet · 2013-08-08T07:57:39Z

With your indication I have found my problem, thanks for that.
But I have a last question.

I only have on elastic the differential between my collection at T time and now. How can I easily synchronise all the collection without use an etl?

richardwilly98 · 2013-08-08T11:36:06Z

The river imports everything available in oplog.rs. In other words everything stored in the collection since replicaset has been enabled.

So you could just do a bulk-update on the collection to get it synchronized in ES. See #47

arnaudroubinet · 2013-08-08T11:53:28Z

Thanks for your help.

arnaudroubinet closed this as completed Aug 8, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No documents found on elastic #111

No documents found on elastic #111

arnaudroubinet commented Aug 7, 2013

richardwilly98 commented Aug 7, 2013

arnaudroubinet commented Aug 7, 2013

richardwilly98 commented Aug 7, 2013

arnaudroubinet commented Aug 7, 2013

richardwilly98 commented Aug 7, 2013

arnaudroubinet commented Aug 7, 2013

richardwilly98 commented Aug 7, 2013

arnaudroubinet commented Aug 8, 2013

richardwilly98 commented Aug 8, 2013

arnaudroubinet commented Aug 8, 2013

No documents found on elastic #111

No documents found on elastic #111

Comments

arnaudroubinet commented Aug 7, 2013

richardwilly98 commented Aug 7, 2013

arnaudroubinet commented Aug 7, 2013

richardwilly98 commented Aug 7, 2013

arnaudroubinet commented Aug 7, 2013

richardwilly98 commented Aug 7, 2013

arnaudroubinet commented Aug 7, 2013

richardwilly98 commented Aug 7, 2013

arnaudroubinet commented Aug 8, 2013

richardwilly98 commented Aug 8, 2013

arnaudroubinet commented Aug 8, 2013