Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OSM import fails with InvalidRecordException: NodeRecord[34] not in use #226

Closed
ehx-v1 opened this issue Apr 4, 2016 · 15 comments
Closed

OSM import fails with InvalidRecordException: NodeRecord[34] not in use #226

ehx-v1 opened this issue Apr 4, 2016 · 15 comments

Comments

@ehx-v1
Copy link
Contributor

ehx-v1 commented Apr 4, 2016

happened during a normal OSMImporter.importFile(...) call with a BatchInserter on an empty graph database:

Exception in thread "main" org.neo4j.kernel.impl.store.InvalidRecordException: NodeRecord[34] not in use
    at org.neo4j.kernel.impl.store.NodeStore.getRecord(NodeStore.java:147)
    at org.neo4j.kernel.impl.store.NodeStore.getRecord(NodeStore.java:138)
    at org.neo4j.kernel.impl.transaction.state.Loaders$1.load(Loaders.java:64)
    at org.neo4j.kernel.impl.transaction.state.Loaders$1.load(Loaders.java:53)
    at org.neo4j.unsafe.batchinsert.DirectRecordAccess.getOrLoad(DirectRecordAccess.java:62)
    at org.neo4j.unsafe.batchinsert.DirectRecordAccess.getOrLoad(DirectRecordAccess.java:39)
    at org.neo4j.kernel.impl.transaction.state.RelationshipCreator.relationshipCreate(RelationshipCreator.java:60)
    at org.neo4j.unsafe.batchinsert.BatchInserterImpl.createRelationship(BatchInserterImpl.java:920)
    at org.neo4j.gis.spatial.osm.OSMImporter$OSMBatchWriter.createRelationship(OSMImporter.java:1554)
    at org.neo4j.gis.spatial.osm.OSMImporter$OSMBatchWriter.createRelationship(OSMImporter.java:1361)
    at org.neo4j.gis.spatial.osm.OSMImporter$OSMWriter.createRelationship(OSMImporter.java:453)
    at org.neo4j.gis.spatial.osm.OSMImporter$OSMWriter.createOSMNode(OSMImporter.java:651)
    at org.neo4j.gis.spatial.osm.OSMImporter.importFile(OSMImporter.java:1928)
    at org.neo4j.gis.spatial.osm.OSMImporter.importFile(OSMImporter.java:1779)
    at com.ptvgroup.haeg.neo4j.experiments.utility.FileImporter.importOSM(FileImporter.java:70)
    at com.ptvgroup.haeg.neo4j.experiments.utility.FileImporter.importFile(FileImporter.java:57)
    at com.ptvgroup.haeg.neo4j.spatialproject.teststuff.TestImport.main(TestImport.java:41)
@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 4, 2016

just found out that already happened in Neo4j but should have been fixed: jexp/batch-import/#65
why does it still appear when working with the OSMImporter, and also on an empty database which should have no relationships, especially without start or end nodes?

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 5, 2016 •
edited
Loading

happened when testing my FileImporter with one-street.osm from the original Spatial project

@ehx-v1 ehx-v1 changed the title InvalidRecordException: NodeRecord[34] not in use OSM import fails with InvalidRecordException: NodeRecord[34] not in use Apr 5, 2016
@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 13, 2016

after re-reading the entry at Neo4j I noticed it doesn't apply to batch inserting, I requested something like putting errroneous relationship records at the end of the import, sorry for troubling you, it's most probably none of your business

@ehx-v1 ehx-v1 closed this as completed Apr 13, 2016
@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 18, 2016

or perhaps YOU could make a queue for data an erroneous relationship would be created from? just a try-catch construct, and in the catch block you put a wrapper for the data in a queue so at the end of the import when all nodes are imported you can try again, so it's only an error if the second node is truly not there (and not just yet to be imported), would this make sense?

@ehx-v1 ehx-v1 reopened this Apr 18, 2016
@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 18, 2016

I have no idea what the OSM data looks like, as I don't get the importer to import the most simple test cases and finding out from the OSM file might get quite hard, so I have no idea how exactly that would look, or even if it's possible at all

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 18, 2016 •
edited
Loading

had this thought because after switching to a normal GraphDatabaseService a quite similar errior appeared:

C:\Users\haeg\Documents\GitHub\spatial(debug output for the previous FileNotFoundException)
Importing with osm-writer: OSMGraphWriter: DatabaseService[Community [C:\Users\haeg\Documents\Neo4j\default.graphdb]]:txInterval[5000]
Exception in thread "main" java.lang.IllegalArgumentException: Other node is null.
    at org.neo4j.kernel.impl.core.NodeProxy.createRelationshipTo(NodeProxy.java:549)
    at org.neo4j.gis.spatial.osm.OSMImporter$OSMGraphWriter.createRelationship(OSMImporter.java:1160)
    at org.neo4j.gis.spatial.osm.OSMImporter$OSMGraphWriter.createRelationship(OSMImporter.java:957)
    at org.neo4j.gis.spatial.osm.OSMImporter$OSMWriter.createRelationship(OSMImporter.java:453)
    at org.neo4j.gis.spatial.osm.OSMImporter$OSMWriter.createOSMNode(OSMImporter.java:651)
    at org.neo4j.gis.spatial.osm.OSMImporter.importFile(OSMImporter.java:1928)
    at org.neo4j.gis.spatial.osm.OSMImporter.importFile(OSMImporter.java:1765)
    at com.ptvgroup.haeg.neo4j.experiments.utility.FileImporter.importOSMAlt(FileImporter.java:83)
    at com.ptvgroup.haeg.neo4j.experiments.utility.FileImporter.importFile(FileImporter.java:57)
    at com.ptvgroup.haeg.neo4j.spatialproject.teststuff.TestImport.main(TestImport.java:41)

that looks to me like an equivalent to the error above, just that the call stack is flattened by that the exception is thrown further upwards

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 18, 2016

here is a screenshot of my test class:
testimport_java
screenshots of my FileImporter can be found at #223 (check the comment right below yours for the latest version)

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 18, 2016 •
edited
Loading

just remembered my FileImporter changed as following:
fileimporter3
the outcommented SuppressWarnings is there because I plan to switch between importOSM and ImportOSMAlt for different test cases, and eventually implement a version that auto-picks the right one depending on the situation (e.g. if it turns out importOSM is better in performance but importOSMAlt covers some error cases importOSM has I would make a try-catch construct with importOSM in try and importOSMAlt in catch), however each of the versions results in one of the described errors

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 18, 2016

BatchInserterManager is just an AutoCloseable wrapping a BatchInserter and shutting it down upon close() so it's basically a mediator between BatchInserter and try-with-resource

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 18, 2016

so as I mentioned, a construct for reading in relationships at the end when necessary would help a lot

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 19, 2016 •
edited
Loading

perhaps I could try adding it too, I just have to figure out how the OSMImporter works, then get Eclipse to recognize Spatial as project, however I might still get done quickly

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 19, 2016

ohh, no, it goes up and down and from inner class out and into the next inner class, I'm too afraid of messing things up to touch this ':-D better you do ;-)

@ehx-v1 ehx-v1 mentioned this issue Apr 19, 2016
Closed
@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 20, 2016 •
edited
Loading

ok I think I should clarify that I plan to use my self-written library in an experimental project for the company I'm having an internship at, so it does have a bit more weight than just some hobby project

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 22, 2016

everything works cleanly when I outcomment the way tag in the OSM but a map without roads is kind of no map right?

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 22, 2016

not sure whether this is still up to date, if you can help me at #232 I don't need this any more, and probably you CAN help me there, so doesn't matter anymore

@ehx-v1 ehx-v1 closed this as completed Apr 22, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ehx-v1