Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSMImporter: BatchInserter and GraphDatabaseService? #223

Closed
ehx-v1 opened this issue Mar 30, 2016 · 8 comments
Closed

OSMImporter: BatchInserter and GraphDatabaseService? #223

ehx-v1 opened this issue Mar 30, 2016 · 8 comments

Comments

@ehx-v1
Copy link
Contributor

ehx-v1 commented Mar 30, 2016

will most probably be labelled "question" I think
but might also be labelled "invalid" because I have no idea whether it belongs here or to the Neo4j forum
does the OSMImporter need the BatchInserter in a way that closes the graph database direction for all GraphDatabaseServices, or can a GraphDatabaseService be open while calling importFile()?
need to know because I'm making some kind of utility library based on Neo4j and Neo4j Spatial, and in this library there's also a FileImporter which can handle both shapefiles and OSM but internally it uses the original Spatial importers
which means, if there can be no GraphDatabaseService running while OSMImporter.importFile() is called I somehow have to pass a user-defined method that creates a (potentially custom) GraphDatabaseService, and I don't want to do that without need because I'm quite fed up with Reflect API after coding ConsistencyManager, so if it does work please let me know that I don't have to mess around with Method objects again (or Runnables, which is not much more convenient)
attached my FileImporter.java (SnippingTool PNG), so you can better understand why it's a problem
fileimporter_java

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Mar 30, 2016

re-thinking it again, there might be a way so it won't be that ugly, making an inner interface and passing that as constructor parameter should be OK

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Mar 30, 2016

the inner interface idea perfectly solved it without a GraphDatabaseService passed in the constructor, sorry for bothering

@ehx-v1 ehx-v1 closed this as completed Mar 30, 2016
@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Mar 30, 2016

test whether a closed issue can still be commented

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Mar 30, 2016

ah ok, it's still possible to comment a closed issue, so if you're interested in my library project ask ahead

@craigtaverner
Copy link
Contributor

I can make a related comment. The OSMImporter can also be called with a normal GraphDatabaseService, instead of the Batch version. This is slower, but has some advantages in being more reliable. The Batch importer is designed for initial import into the database, and is not ideal for importing at any point later on. I believe there are risks in doing that.

I am certainly interested in your library project. Interesting to import shapefiles and OSM to the same database.

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 1, 2016

btw now the used GraphdatabaseService is returned
here the new version (2 screenshots because it doesn't fit on one screen anymore):
fileimporter1
fileimporter2
Javadoc for setDatabaseDirectory is WIP, also I hope "thrown by methods called by this method" someday can be replaced with particular cases of throw (that's where you come in, see my Javadoc issue)

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 1, 2016

so if you want to import mltiple files to the same database you should go for fileImporter.importFile().shutdown() each time (except for the last, especially if you want to make more stuff on the database)
right now the other classes in the lib are ConsistencyManager (an abstract GraphDatabaseService class wrapping around another GDBS and allowing for defining consistency terms, e.g. conditions under which nodes or relationships can be made, or fixed properties for nodes and relationships which must not be removed and can be initted any way you like to, I'll link to my GitHub repo for the library project once it exists so if you like to you can change your SpatialDatabaseService to inherit from ConsistencyManager which is quite an independant class, btw made before I looked at Spatial for the first time), a BatchInserterManager (an AutoCloseable wrapping around BatchInserters and shutting them down upon close(), which allows for try-with-resource) and a DatabaseShutdownHookProvider (creates shutdown hooks for GDBSes and also avoids duping shutdown hooks, because I was unsure what happens with a down GDBS shutdown() is called at but even if this is safe it helps against overflooding the heap with shutdown hooks), I'm also thinking whether I should build in a DatabaseTransferOperator with tools for working at multiple GDBSes for different locations at the same time (e.g. migrating data from one GDBS to another)
but I think before starting with the DatabaseTransferOperator I should get done with Javadoc first (I don't like documenting much more than you but I just feel like it's a duty of every programmer because it makes the library easier to use)
ah and almost forgot to clarify this is NOT April Fools ;-)

@ehx-v1
Copy link
Contributor Author

ehx-v1 commented Apr 1, 2016

and there are some Neo4j-unrelated classes which can also be (more or less) useful, 2 of which (AdvancedList and NotApplicableException) are even used within the ConsistencyManager
the AdvancedList is inheriting from the ArrayList without changing implementation, however containing some useful operations I once saw no point in implementing again and again, so I made that class, and later I started thinking about which problems there might be with the methods the way they were made and on spots where I had no idea how to fix it I just made a new method so users can deal better with it
the NotApplicableException is... partially a legacy exception from before I knew that there is such thing as UnsupportedOperationException, and when my Eclipse kept claiming IllegalArgumentException is a checked exception, and the trouble is you can't use checked exceptions in framework classes that were not intended for them to be thrown (except by messing around with Reflect API, which I hadn't come in touch with before making ConsistencyManager) so I needed a custom exception because no (to my Eclipse) unchecked exception was fitting, however later on I found a context for NotApplicableException which perfectly fits it and, as far as I know, is not covered by any Exception class yet: argument and state are incompatible - so that's what it now should be used for
the VerboseWrapperException is... well, a bit shaming to admit it, but... so far a fail. I actually wanted an exception that wraps around another exception and behaves as though it was the original exception, however makes its throw printout more verbose (escpecially referring to SQLExceptions), but I can't seem to make the actual throw output change...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants