Updates no longer work with Nominatim 4.x

[lonvia]

Nominatim has moved some of the indexing processing into Python code since version 4.x. The simple update mechansim that Photon uses thus only does half the work. In particular, addr:* parts a no longer properly matched.

If you run regular update on your Photon database you must remain with Nominatim 3.7 for now.

Imports still work against the newer version.

[yakimka]

Hello! Is it possible to fix it somehow, or updates will never work with version >=4

[lonvia]

There will probably be a new update mechanism at some point but it isn't exactly a priority at the moment.

[yakimka]

Can you help with advice on updating the index without downtime if I am using Nominatim 4?

[mtmail]

That's currently not possible with Nominatim v4.

[RhinoDevel]

That's currently not possible with Nominatim v4.

Sorry to bother, but is the updating/indexing with Photon v0.4.3 and Nominatim version > 3.7 still not possible?

Thanks in advance!

[geocept]

I am running photon 0.4.3 and nominatim 4.3.1.

Nominatim data is kept uptodate by running this once a day: nominatim replication --once

Photon call "curl http://localhost:2322/nominatim-update" returns "404 Not found".

Is it my fault or is update from noninatim data still not possible. If it is not possible you should delete nominatim updates in your documentation and delete the nominatim update script from repository. Like this it is very missleading.

[lonvia]

The issue is still open.

[ybert]

I could try to give it some time in the next weeks.
But I need a more precise idea of what is not working in the update and what is missing ?

I never used it as I managed zero downtime update by having a load balancer with health-checks and two photon servers behind. With this setup you can stop one photon service, import full data and restart the service. Then you can do the same on the second photon service.

[lonvia]

Nominatim updates used to work this way: 1. Import updates using osm2pgsql, 2. run a single SQL command UPDATE placex SET indexed_status = 0 WHERE indexed_status > 0. Photon simply would inject itself during that second step, read the changes made by the SQL command and insert it in its own database. The second step now has changed. There is some pre-processing done in the Python code. So you forcibly must run nominatim index or the search index is built up in the wrong way.

I see two ways to solve the problem:

1. Extend nominatim index to emit the changed data while it processes them as a generic JSON file and import that into Photon. The output should be generally usable and not exactly tailored to Photon, also to make sure the two projects remain independent.
2. Adapt Photon to find changes using the indexed_date column. Then the updates can be run after nominiatim index is finished.

The first solution is more generic but the second one likely easier to implement. So I was leaning towards the second one for now.

[geocept]

ybert: at the moment stoping service and importing full data is what we do too.
As photon is not used on weekend and has other backups we can live with short downtime.

A cron jobs runs this script once a week and creates a new data extract from our daily updated nominatim database:

`
cd /mnt/overpass_api/photon/photon_data_import
rm -R photon_data

java -jar /mnt/overpass_api/photon/photon-*.jar -nominatim-import -host localhost -port 5432 -database nominatim -user photon -password 'xxxxxxxxxxxx' -languages de,en,fr

systemctl stop photon.service
mv photon_data_latest photon_data_old
mv photon_data photon_data_latest
chown -R nominatim:nominatim photon_data_latest
systemctl start photon.service
`

Creating new full nominatim import for all europe takes aprox 24h on our small server. And after it finishes restart with new data is done quickly. It works for us.
But of course this solution is not very nice. It is a waste of 100GB storage for storing current and old data extract. And it wasts cpu and ram resources for new extracts.

Would be great of one of you would reenable the "live" updates.

Thanks for great work on photon and nominatim btw.