Support arbitrary languages

[lonvia]

Photon should be able to do searches over all available language variants mapped in OSM (i.e. all name:* tags). Bonus points if it can improve search results by guessing the language of the query right and re-weight the results accordingly.

[tommedema]

Note that photon cannot assume that western scripts are inserted, e.g. Chinese, Arabic or Thai should work fine as well (Nominatim supports this too).

[christophlingg]

how do you guess the user language in nominatim? by analyzing http headers / ip address?

[tommedema]

Nominatim accepts an accept-language parameter. If that is not set, I believe it looks at the browser's header of accepted languages, but I didn't verify this.

[lonvia]

For nominatim: accept-language parameter first and then HTTP accept-language header but this is used only for selecting the right name:* to output. It is not used in any way to rank the query matches or results.

[christophlingg]

Thanks @lonvia !

[karussell]

Language detection could be done via a simple hack I made some years ago or other tools like described here

[christophlingg]

sorry, wrong ticket ;-)

[karussell]

My comment here is relevant in this ticket as well. I'll hopefully have time in the next days to try that.

[karussell]

I am creating a language detector specific for 'local' names like street names and POIs - i.e. the detector learns from OpenStreetMap data itself. Look here

It is already good for German stuff (4% of German names not detected as German), but e.g. french and english detection is really bad (30% error). I'll see how I can improve this.

Update 2020: there is a new tool https://github.com/pemistahl/lingua

[christophlingg]

One of the biggest problems yet to resolve for full multilingual support is the storage size. The current elasticsearch config let size grow linear with the number of languages (approx 30 GB each).

I hope the cross_field approach mentioned previously can help.

[harry-wilson]

@lonvia , using Photon for customized data on Nominatim database does not support any languages other than en, fr, de, it. How can we search results by languages such as Arabic?