Use a GDAL+PROJ (from GMT) build with VisualStudio

[joa-quim]

Hi,
I'm not really sure how to proceed with this one. Ideally it should be a PR but it probably involves more knowledge on julia building pacakges that I have. So I provide it as an example on how we can make GDAL.jl use the full GDAL (3) build that comes with the GMT installer for Windows. Currently in a RC4 stage but soon an official release.

After installing it Windows user have a rather complete GDAL3+PROJ6.2 package ready available for use. So Windows julia users would benefit if they could use it instead of installing an extra 500 Mb package that has less drivers.

As example I modified GDAL.jl (attached) to use that GDAL version, but only if it's found via path. Besides this it's also necessary to edit test\gdal_utils.jl and wrap the line 168 with a try block because the GMT+GDAL build does not create .aux.xml files by default

try
    rm("data/utmtiny.asc.aux.xml")
catch
end

with that the tests pass

(v1.1) pkg> test GDAL
   Testing GDAL
    Status `C:\tmp\jl_83F.tmp\Manifest.toml`
  [b99e7846] BinaryProvider v0.5.6
  [add2ef01] GDAL v0.2.0
  [2a0f44e3] Base64  [`@stdlib/Base64`]
  [8ba89e20] Distributed  [`@stdlib/Distributed`]
  [b77e0a4c] InteractiveUtils  [`@stdlib/InteractiveUtils`]
  [8f399da3] Libdl  [`@stdlib/Libdl`]
  [56ddb016] Logging  [`@stdlib/Logging`]
  [d6f4376e] Markdown  [`@stdlib/Markdown`]
  [9a3f8284] Random  [`@stdlib/Random`]
  [ea8e919c] SHA  [`@stdlib/SHA`]
  [9e88b42a] Serialization  [`@stdlib/Serialization`]
  [6462fe0b] Sockets  [`@stdlib/Sockets`]
  [8dfed614] Test  [`@stdlib/Test`]
┌ Info: GDAL 3.1.0dev, released 2019/99/99
│ 220 GDAL drivers found
└ 84 OGR drivers found
   Testing GDAL tests passed

It would be nice if this solution could be integrated in the GDAL.jl official package, but in case you agree I will need you help to finish it.

Joaquim

GDAL.zip

[visr]

If I understand correctly, this means GDAL.jl would automatically use a different binary depending on if GMT happens to be installed. That would be taking the opposite direction that the julia community is going. Quoting Elliot from here: "Death to all global mutable state!" :)

I understand that it is frustrating that the current default provided binary is rather minimal whilst you have a perfectly fine GDAL already installed, but I hope you see how much easier it makes things for us if we know exactly what we provide to users. Starting from Julia 1.3 we will also have a much better system for binary artifacts integrated into the package manager, so we won't have to duplicate dependencies anymore either between packages. This work should also make it easier to add more drivers to the default install, see also #65.

As I mentioned before in https://discourse.julialang.org/t/package-size-crazziness/14644/14, it is perfectly fine to "bring your own GDAL" and use that (with no guarantees), but I'd rather not have the default depend on what happens to be on the computer.

[joa-quim]

Yes, you got it right. But it would be on Windows only. And I wouldn't say that if "what happens to be in the computer" is either a GMT or Gisinternals or OSGeo4Win GDAL build, is a risk. On the contrary.

What I dislike in the BinaryProvider is that it forces the use of MinGW binaries. I have seen cases where those binaries are up to 5 times larger than VS ones (but I'm saying that's always the case) and if one needs to debug them, there is no comparison on using VS and dbg.

But maybe there is hope, from one link in this discourse post (I'm not alone in this wish) is seems that it will be possible to choose which version to install. As long as that will be easy for users to select, this would be a good solution.

[visr]

Yeah no my point about "what happens to be installed" is not about security. It's that it becomes impossible to control the binary. Now we know exactly what users are getting on a default install. If we make it dependent on what happens to be installed, it will be hard to help users, as I have no idea what they are running. I have nothing setup in this package right now to handle multiple GDAL versions for instance. Also if they remove their GMT install and as a result GDAL.jl will stop working that may be unexpected. So that's why I don't want to go there as a default setting.

Indeed the new artifacts system will probably make it easier to point to your own install, that's great.

[visr]

I think I'll close this for now, since I think it's better not picking up existing GDAL installs by default.

Some documentation for how to make it point to other installs would be welcome. Right now this is means changing the paths in the deps/deps*.jl files. Though that should become easier once we start using the Pkg artifact system.

[joa-quim]

OK, but the issue is that it's not only a question of documentation. Because the GMT installer comes with a full GDAL3+PROJ6.2 the the GDAL.jl that I posted has provisions to add the GDAL_DATA and PROJ_LIB env variables pointing to the GMT instalation using the GDAL's new localpaths.

[visr]

Yeah I understand that the issue was not about documentation, and that the GMT installer provides a nice and complete GDAL install that GDAL.jl can use. But this issue is about making that the default, but I don't want to make it the default, for the reasons I gave above. I'd rather have some help to improve the default GDALBuilder install. Reason I am not working on it right now is because it is about to change in major ways (JuliaPackaging/BinaryBuilder.jl#441), and I prefer to invest my limited time in the new Pkg artifact system.

[joa-quim]

Perhaps there is a little misunderstanding here. I didn't intend to make it the default install for Windows (for example the gisinternals is probably even more complete than mine). That was one of the reasons why I didn't make this as a PR. But taking for example the PyCall, they provide a mechanism to say don't reinstall Conda, I already have one or a python install. It's something of this kind that I was thinking as a nice to have in the GDAL.jl package, but I know little on the julia building systems.

Regarding the BinaryBilder, when it will be possible to build with VS I can try to help for the packages that I already build (and know how to).

[visr]

Yeah I see. By "default" I meant not use GMT by default, but pick up GMT if it is found, with a default install as well. But I'd definitely want behavior like that to be opt-in, and not default.

However, with the Pkg artifacts system around the corner (1.3 is now on RC2), it seems better to wait and use the Pkg override system, instead of building our own solution that will be obsolete so quickly.