Very slow geocoding after following installation instructions

[kannes]

I followed https://github.com/komoot/photon#installation

My system's decent: 3rd generation i5, 20G RAM, the photon_data directory is on a HDD that hdparm reads about 200MB/s on.

But the queries take many seconds to return. 3 seconds seems to be the minimum.

$ time curl -s "http://localhost:2322/api?q=berlin" > /dev/null
real 0m2.995s
user 0m0.006s
sys 0m0.003s

$ time curl -s "http://localhost:2322/api?q=berlin&lon=10&lat=52" > /dev/null
real 0m49.391s
user 0m0.009s
sys 0m0.003s

$ time curl -s "http://localhost:2322/reverse?lon=10&lat=52" > /dev/null
real 1m22.922s
user 0m0.005s
sys 0m0.003s

$ time curl -s "http://localhost:2322/api?q=berlin&lang=it" > /dev/null
real 0m2.997s
user 0m0.007s
sys 0m0.003s

$ time curl -s "http://localhost:2322/api?q=berlin&limit=2" > /dev/null
real 0m2.727s
user 0m0.004s
sys 0m0.002s

How can I get reasonably fast results?

[systemed]

Try an SSD?

[kannes]

Ok!

$ time curl -s "http://localhost:2322/api?q=berlin" > /dev/null
real 0m3.217s

$ time curl -s "http://localhost:2322/api?q=berlin&lon=10&lat=52" > /dev/null
real 0m18.666s

$ time curl -s "http://localhost:2322/reverse?lon=10&lat=52" > /dev/null
real 0m45.795s

$ time curl -s "http://localhost:2322/api?q=berlin&lang=it" > /dev/null
real 0m2.885s

$ time curl -s "http://localhost:2322/api?q=berlin&limit=2" > /dev/null
real 0m3.064s

Still all at least 3 seconds. The long ones got more than twice as fast but still ... 45s?

(For the record, I always restarted photon between each query to make sure there is no immediate hot cache. I did wait a sufficient amount of time before each query.)

[karussell]

What is your RAM setting for ElasticSearch and/or photon? Try to reduce the -Xmx setting so that more off-heap memory is available for ElasticSearch.

[kannes]

I run java -jar photon-0.2.7.jar as instructed by the installation guide. I have no idea where to set ES specific options.

I now tried java -Xms6G -Xmx12G -jar photon-0.2.7.jar, that did not change much. The CPU is fully utilized when running the queries.

Guess I am too spoiled by graphhopper's performance out of the box...

Here is a log if that helps finding the problem?

$ java -jar photon-0.2.7.jar
[main] ERROR de.komoot.photon.elasticsearch.Server - cannot install plugin: mobz/elasticsearch-head: java.io.IOException: plugin directory /tmp/photon/photon_data/elasticsearch/plugins/head already exists. To update the plugin, uninstall it first using --remove mobz/elasticsearch-head command
[main] ERROR de.komoot.photon.elasticsearch.Server - cannot install plugin: polyfractal/elasticsearch-inquisitor: java.io.IOException: plugin directory /tmp/photon/photon_data/elasticsearch/plugins/inquisitor already exists. To update the plugin, uninstall it first using --remove polyfractal/elasticsearch-inquisitor command
[main] INFO org.elasticsearch.node - [Bram Velsing] version[1.7.0], pid[23646], build[929b973/2015-07-16T14:31:07Z]
[main] INFO org.elasticsearch.node - [Bram Velsing] initializing ...
[main] INFO org.elasticsearch.plugins - [Bram Velsing] loaded [wordending], sites [inquisitor, head]
[main] INFO org.elasticsearch.env - [Bram Velsing] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [69.4gb], net total_space [439.1gb], types [ext4]
[main] WARN org.elasticsearch.bootstrap - JNA not found. native methods will be disabled.
[main] WARN org.elasticsearch.script - [Bram Velsing] deprecated setting [script.disable_dynamic] is set, replace with fine-grained scripting settings (e.g. script.inline, script.indexed, script.file)
[main] INFO org.elasticsearch.script - [Bram Velsing] compiling script file [/tmp/photon/photon_data/elasticsearch/config/scripts/general-score.groovy]
[main] INFO org.elasticsearch.script - [Bram Velsing] compiling script file [/tmp/photon/photon_data/elasticsearch/config/scripts/location-biased-score.groovy]
[main] INFO org.elasticsearch.node - [Bram Velsing] initialized
[main] INFO org.elasticsearch.node - [Bram Velsing] starting ...
[main] INFO org.elasticsearch.transport - [Bram Velsing] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[/127.0.0.1:9300]}
[main] INFO org.elasticsearch.discovery - [Bram Velsing] photon/Atf9kGAPSziG-xFyKbCzPg
[elasticsearch[Bram Velsing][clusterService#updateTask][T#1]] INFO org.elasticsearch.cluster.service - [Bram Velsing] new_master [Bram Velsing][Atf9kGAPSziG-xFyKbCzPg][hilmar][inet[/127.0.0.1:9300]], reason: zen-disco-join (elected_as_master)
[main] INFO org.elasticsearch.http - [Bram Velsing] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/127.0.0.1:9200]}
[main] INFO org.elasticsearch.node - [Bram Velsing] started
[main] INFO de.komoot.photon.elasticsearch.Server - started elastic search node
[elasticsearch[Bram Velsing][clusterService#updateTask][T#1]] INFO org.elasticsearch.gateway - [Bram Velsing] recovered [1] indices into cluster_state
== Spark has ignited ...
>> Listening on 0.0.0.0:2322
[Thread-1] INFO org.eclipse.jetty.server.Server - jetty-9.0.z-SNAPSHOT
[Thread-1] INFO org.eclipse.jetty.server.ServerConnector - Started ServerConnector@6716bac2{HTTP/1.1}{0.0.0.0:2322}

[karussell]

Can you try -Xms12g -Xmx12g? And then run 3 warm up queries and after this it should be fine.

[kannes]

Ah, seems like patience is key. Thanks! I managed to get ~25 queries/s by simply running through addresses of a specific city and letting it start slow.

Does it start caching in some kind of hierarchy?

[kannes]

Well, it did not become super fast, but simple "warming" resolved the very slow queries so I am closing this.

[stouch]

I meet this issue too (#726)

The strange thing is that for a same request it can take 2sec, 200ms or 90sec.