[QMS-571] MacOS build scripts and CMakeLists.txt changed

[d029940]

What is the linked issue for this pull request:

QMS-#571

What you have done:

Changed build files und MacOSX dir in order to

• create an Intel version of QMS
• ARM and Intel packages which rely on brew packages at run time (prep for a brew QMS package)
• Changed Mac Source files in order to make QMapTool working

Steps to perform a simple smoke test:

1. create an empty folder and cd in this folder
2. clone d029940 QMS repo and switch to QMS-571 branch
3. run ./build-all.sh in MacOSX dir
4. check QMapShack / QMapTool are built (self contained app)
5. run ./build-all.sh -b in MacOSX dir
6. check QMapShack / QMapTool are built (brew dependant app)

Does the code comply to the coding rules and naming conventions Coding Guidelines:

• yes

Is every user facing string in a tr() macro?

• yes

Did you add the ticket number and title into the changelog? Keep the numeric order in each release block.

• yes, I didn't forget to change changelog.txt

[mitxel-m]

Thank you very much for all this work. I don't own a mac and I can't test it, but I have taken a look at the scripts and the gdal compile step has caught my attention.

For self-compiled gdal: In the configure options I can't see any reference to libcurl and sqlite

• sqlite is needed to read some map formats like .MBTILES which are quite common nowadays.

• curl is needed to connect to some online DEM servers, and maybe also for some online map servers.

Without that support qmapshack also works, but its usability decreases.

For gdal brought by homebrew: Maybe it already comes with sqlite and curl support and nothing else is needed.

I think it's worth to test if the final bundle is able to open this kind of files, but I don't have a mac to do it myself. At least to make it easier I have prepared some ready to use files, if anyone wants to do a test: download 1.2 Mb

Excuse me if this is already solved and this comment is just noise.

---

For reference

MBTILES: https://gdal.org/drivers/raster/mbtiles.html

GDAL/OGR must be compiled with OGR SQLite driver support, and JPEG and PNG drivers.

WCS: https://gdal.org/drivers/raster/wcs.html

WMTS: https://gdal.org/drivers/raster/wcs.html

Access to WMTS layers is possible with the GDAL WMTS client driver (needs Curl support).

Compile gdal from source with curl and sqlite:

https://gdal.org/development/building_from_source.html#sqlite3

https://gdal.org/development/building_from_source.html#curl

[d029940]

It is always good to get a feedback.
curl and sqlite3 comes with the MacOS.
But I figured out that I have to switch on the GDAL_USE_EXTERNAL_LIBS. This is apparently the default.
As far as I understand the brew formula of GDAL, they run cmake with no additional options.
So, your advice is welcome.
Currently the released QMapShack.app and QMapTool.app have been built with the brew package.

And I am not sure how QMapShack and QMapTool are using GDAL.
Well, QMapShack is using gdalbuildvrt and QMapTool has references of gdaladdo, gdal_translate, gdalwarp and gdalbuildvrt GDALin the setup menu.

If you are interested I uploaded the log of the GDAL build.
log.txt

[kiozen]

All tools should be provided (afaik that is already the case). The rest are plugins. If libraries are found the modules are compiled. Usually the more of the optional requirements are present the better as more file formats can be used by QMapShack. This is true for all raster map formats. QMS/QMT do not make use of GDAL's vector map support. (https://gdal.org/drivers/vector/index.html). So no need to spend time on these.

[d029940]

Thank you for the comment.
For the next PR (not this one), should I enable GDAL_USE_EXTERNAL_LIBS?

Anyway, this is more academic. Building GDAL from source is only a fallback solution. And only interesting, if I want

1. to deploy QMS, where on the user side the brew packages need to be installed (instead of having them bundled) AND
2. to avoid to install GDAL with brew, because it ties in a bunch of dependencies.

But up to now, I am relying on the GDAL package from homebrew.

[kiozen]

GDAL_USE_EXTERNAL_LIBS is more about using internal code vs external libs, e.g. libgeotif vs internal implementation. I would leave it as the default says.

[mitxel-m]

@d029940 On your log.txt all seems to be OK for curl and sqlite support.

Apologies if this was just noise. Since gdal 3.5 and with cmake the way of passing options when compiling gdal has changed. In the old way the --with-sqlite3=yes option was a classic in all recipes (I assumed it was mandatory). Now, as noted in the wiki, this option is set to ON by default, and gdal compiles with curl and sqlite3 if they are found in the system. So if sqlite3 and curl are already on MacOS there is no problem.

[wthaem]

@mitxel-m: (Slightly off-topic): I compiled QMS on Windows10 with GDAL 3.6.1 using the new cmake build process. WCS driver is properly installed and WCS files seem to work with GDAL (gdaldem.exe). But using your or other WCS files in QMS to get DEM info fails. I observed this already with some earlier GDAL version. Did you try the newest GDAL with WCS in QMS and did it work? Any hint would be appreciated. Thanks.

[d029940]

If I build QMS myself on the Mac, I do a git clone https://github.com/OSGeo/gdal.git. So, I think it is the most current version.
But GDAL building myself is only a fallback solution. The normal build uses the GDAL package from Homebrew. And so is the build of the binaries in https://github.com/d029940/qmapshack/releases/tag/QMapShack-Mac.
If you want to build QMS on a Mac yourself, I would rather advise you to clone my fork of QMS and switch to the "work" branch. (this has the latest updates to the GDAL build discussed in this PR.) Follow the instructions and use the -g flag parameter, i.e. build-all.sh -g
Since, I am not familiar with GDAL and how QMS is using it, WCS, ... are unknown to me.

[mitxel-m]

@wthaem some hints (one of them also related to MacOs ):

1. the url indicated in the .wcs file may be out of service. Sometimes the server url is changed by the authorities.

2. it may be a certificate problem.

We have seen this problem on a friend's mac, with MacOs Monterey and running QMS installable version 1.15.2.
It worked for wcs files pointing to urls of type http: but failed for those pointing to https:.

By chance I was able to meet my friend this past week and we managed to try launching QMS in a terminal, to get the error message.

Although the mac application bundle already contained the curl libraries and executable, it was asking for a .crt certificate file in a system folder that didn't exist.
/opt/local/share/curl/curl-ca-bundle.crt

Once we created the folder and put the curl-ca-bundle.crt there everything worked correctly.

We got the .crt file from curl website:

https://curl.se/docs/caextract.html

by downloading cacert.pem and renaming it to curl-ca-bundle.crt.

Remark the notes about the file name at curl website:

File name

Some programs will expect this file to be named ca-bundle.crt (in the correct path).

curl on windows has a system to find it if named curl-ca-bundle.crt

With this in mind when I have seen @wthaem question I have inspected a laptop with win10 and QMS 1.16.1. on which wcs works fine.
Surprise: there is no curl-ca-bundle.crt in the qmapshack directory. However, there are both copies in folders of two other applications that use gdal+curl: QGIS and the old QlandkarteGT. In both cases in the same folder where curl.exe is located.

@wthaem Maybe you can try that. Alternatively you can also try using a a GDAL configuration variable that tells the curl library to skip SSL verification.

set GDAL_HTTP_UNSAFESSL=YES
qmapshack

@d029940 (sorry if all this is a bit off-topic) For the purpose of this PR, I suppose it is not necessary to provide this .crt file, probably gdal and curl provided by brew will use certificates that are already in the system, but I am not sure. A simple test would be to try the online DEM provided by QMS: `World Online SRTM900'.

[wthaem]

@mitxel-m: Thanks for the hints!

it may be a certificate problem:

It was a certificate problem! I copied the missing curl-ca-bundle.crt to my files directory and everything was fine again.

[d029940]

I found this https://blog.bytesguy.com/resolving-lets-encrypt-issues-with-curl-on-macos

If you already have brew installed, then installing curl and libcurl will solve the problem, given that the "brew" curl is found in the path before the "MacOS" curl.

Do you have a test case, so that I can check it myself?

[mitxel-m]

@d029940

Do you have a test case, so that I can check it myself?

Sure

• You can try the online DEM provided by QMS: `World Online SRTM900'.
• You can also try the Spanish IGN online public DEM download test files -1.2Mb

If you are not used to online DEMS: remember to activate the DEM online only when you are already in a close zoom level. It is also highly recommended to limit its range of use with the slider (similar to maps). By doing so, you avoid to request millions of points from the server at outer zoom levels, and QMS will work smoothly. Once you have limited the range of use for DEM, doing zoom out of that range the DEM remains in a kind of silent or background mode. This is, the DEM data will not be displayed on screen, but data will be applied when you draw a track.

[d029940]

Thank you for the test cases. I took both and I did not have any problems activating the wcs files. DEM data are shown.
Should I have a problem with certificates? The QMS-App installed (1.16.1) does not contain curl. So, if curl is used, then QMS ran the MacOS curl.

Am I on the wrong track?

[mitxel-m]

Thank you for the test cases. I took both and I did not have any problems activating the wcs files. DEM data are shown. Should I have a problem with certificates? The QMS-App installed (1.16.1) does not contain curl. So, if curl is used, then QMS ran the MacOS curl.

Am I on the wrong track?

My knowledge about curl is very basic, but I think you have taken the right track. That way it will be more difficult to have a certificate problem, assuming that the system will take care to update them periodically.

Otherwise, a certificate file included in an installation bundle may become obsolete when some certificates expire. You will probably be able to replace it manually, but a regular user could not be aware of that. _ Not a big problem assuming a regular user will read the wiki to find how to fix it..._ :-)

@wthaem thanks for updating the wiki so fast with instructions on how to solve this !

[d029940]

Just a question to kiozen:

I am not sure, If I need to release this PR explicitly?

[kiozen]

What do you mean by "release"? If you want to have it merged to dev simply give me a ping that all concerns are settled and there will be no other change. Then I will do a few smoke tests and if nothing pops up I merge it.

[d029940]

The branch can be merged. After pushing some cosmetic changed to my repo, I did a last successful build and test (as far as I am using QMS).

I also added some information to https://github.com/Maproom/qmapshack/wiki/TroubleShooting, but I do not know how to update the TOC.

And how to deal with the binaries on https://github.com/d029940/qmapshack/releases/tag/QMapShack-Mac?
Refer to them from the official download page?

[mitxel-m]

For testing it is Ok pointing to the fork,
Once merged to dev, should it point to the main repo or keep pointing to the fork ?

clone git repo https://github.com/d029940/qmapshack.git
or
clone git repo https://github.com/Maproom/qmapshack.git

[d029940]

Good point. Just updated it.

[wthaem]

@d029940:

I also added some information to https://github.com/Maproom/qmapshack/wiki/TroubleShooting, but I do not know how to update the TOC.

Don't worry! I'll update the TOC for you soon! - Done!