Builds are broken with the latest s3 bucket change.

[ardell]

Starting this afternoon I'm getting the following when trying to deploy using this buildpack https://github.com/cyberdelia/heroku-geo-buildpack.git#1.3

remote: Building source:
remote:
remote: -----> geos/gdal/proj app detected
remote:        Using geos version: <?xml version="1.0" encoding="UTF-8"?>
remote:        Using gdal version: <?xml version="1.0" encoding="UTF-8"?>
remote:        Using proj version: <?xml version="1.0" encoding="UTF-8"?>
remote: /app/tmp/buildpacks/ea0f857e95a9e05a1eb6c6e01735411a11c435d901109be7dad253990648880dfe2899f5f76e7ce5c00ad0b409d39adce0ba318996095093401ea51391311c0b/bin/compile: line 73: [: too many arguments
remote: /app/tmp/buildpacks/ea0f857e95a9e05a1eb6c6e01735411a11c435d901109be7dad253990648880dfe2899f5f76e7ce5c00ad0b409d39adce0ba318996095093401ea51391311c0b/bin/compile: line 79: [: too many arguments
remote: /app/tmp/buildpacks/ea0f857e95a9e05a1eb6c6e01735411a11c435d901109be7dad253990648880dfe2899f5f76e7ce5c00ad0b409d39adce0ba318996095093401ea51391311c0b/bin/compile: line 85: [: too many arguments
remote: -----> Vendoring geo libraries done

[ardell]

From what I can gather it looks like the manifest files were moved from cyberdelia-geo-buildpack (no such bucket), but permissions are wrong on the new bucket heroku-buildpack-geo (access denied).

Old:
http://cyberdelia-geo-buildpack.s3.amazonaws.com/cedar-14/manifest.geos
http://cyberdelia-geo-buildpack.s3.amazonaws.com/cedar-14/manifest.gdal
http://cyberdelia-geo-buildpack.s3.amazonaws.com/cedar-14/manifest.proj

New:
http://heroku-buildpack-geo.s3.amazonaws.com/cedar-14/manifest.geos
http://heroku-buildpack-geo.s3.amazonaws.com/cedar-14/manifest.gdal
http://heroku-buildpack-geo.s3.amazonaws.com/cedar-14/manifest.proj

[zerkz]

We just had the same thing happen to my company, during a prod deploy.

[cyberdelia]

I had to change the bucket in the hurry, and dropped support for deprecated stacks.

[ardell]

@cyberdelia could you either grant public access to the manifest files on S3 or publish them elsewhere?

We're not able to deploy until this is fixed.

[cyberdelia]

@ardell Are you unable to use the heroku-16 stack?

[ardell]

@cyberdelia I have not tested on cedar-14 yet, and would like to be able to deploy if anything comes up in the meantime.

I understand not wanting to support cedar-14 indefinitely but is there something we can do short-term until we get migrated?

[alexey-krasilnikov]

I get the same error on heroku-16

remote: -----> geos/gdal/proj app detected remote: Using geos version: <?xml version="1.0" encoding="UTF-8"?> remote: Using gdal version: <?xml version="1.0" encoding="UTF-8"?> remote: Using proj version: <?xml version="1.0" encoding="UTF-8"?> remote: /app/tmp/buildpacks/3174f4234c5151450fcaba5f50a050126498959c8209a7a6057230bfb29be2eaef3ae0098ab726ef807728f7b4792e31db50a2b89636ae181d9e71b03f2f83c1/bin/compile: line 82: [: too many arguments remote: /app/tmp/buildpacks/3174f4234c5151450fcaba5f50a050126498959c8209a7a6057230bfb29be2eaef3ae0098ab726ef807728f7b4792e31db50a2b89636ae181d9e71b03f2f83c1/bin/compile: line 88: [: too many arguments remote: /app/tmp/buildpacks/3174f4234c5151450fcaba5f50a050126498959c8209a7a6057230bfb29be2eaef3ae0098ab726ef807728f7b4792e31db50a2b89636ae181d9e71b03f2f83c1/bin/compile: line 94: [: too many arguments remote: cp: target '/tmp/build_d200a9439d8446e031c3a245dff9b35d/.heroku/vendor/lib/.' is not a directory remote: ! Push rejected, failed to compile geos/gdal/proj app.

[ardell]

@scurramalum Try bumping the version of heroku-geo-buildpack from 1.3 to 1.4.

Instead of https://github.com/cyberdelia/heroku-geo-buildpack.git#1.3 you want just https://github.com/cyberdelia/heroku-geo-buildpack.git#1.4

heroku buildpacks
heroku buildpacks:remove 'https://github.com/cyberdelia/heroku-geo-buildpack.git#1.3'
heroku buildpacks:add 'https://github.com/cyberdelia/heroku-geo-buildpack.git#1.4'

You may need to use the --index flag on the buildpacks:add command to get it back in the same order it was in before.

[ardell]

@cyberdelia I tried upgrading to heroku-16 stack and the manifest files now work, but now I'm getting:

irb(main):001:0> RGeo::Geos.supported?
=> false
irb(main):002:0> RGeo::CoordSys::Proj4.supported?
=> false

Whereas production is giving me true for both of these.

Here's the output from gem install rgeo --verbose ...

$> gem install rgeo --verbose
...
Building native extensions.  This could take a while...
current directory: /app/vendor/ruby-[version]/lib/ruby/gems/[version]/gems/rgeo-1.1.1/ext/geos_c_impl
/app/vendor/ruby-[version]/bin/ruby -r ./siteconf20180812-10-1gjvkqq.rb extconf.rb
checking for geos-config... no
checking for geos_c.h... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Any ideas?

[denishaskin]

Ouch. Summarily dropping support for deprecated stacks seemed a bit harsh. And we've now lost a significant part of production functionality during a routine deploy and it's unclear (yet) what the best path is to recover. goes to brew a large pot of coffee

[denishaskin]

@ardell fwiw, I just did the upgrade to heroku-16 and deployed, and things seem okay for us.

irb(main):001:0> require 'rgeo'
=> false
irb(main):002:0> RGeo::Geos.supported?
=> true

And our code that was failing before this is now working again.

pours rest of coffee down sink

[sheerun]

It seems s3 files has been removed by author. We need either someone to post mirror repository or everyone needs to switch to newest heroku stack immediately..

[ardell]

@denishaskin that's encouraging, thanks!

What version of the rgeo gem are you running? Wondering if I'll need to upgrade that as well...

[denishaskin]

@ardell We have no version specified in the Gemfile.

Various relevant bits from the build log:

-----> geos/gdal/proj app detected
       Using geos version: 3.6.1
       Using gdal version: 2.1.3
       Using proj version: 4.9.3
-----> Fetching and vendoring geos
-----> Fetching and vendoring gdal
-----> Fetching and vendoring proj
-----> Vendoring geo libraries done
...
       Fetching rgeo 0.6.0
       Installing rgeo 0.6.0 with native extensions
...
       Fetching rgeo-geojson 0.4.3
       Installing rgeo-geojson 0.4.3
...
       Fetching rgeo-activerecord 5.1.1
       Installing rgeo-activerecord 5.1.1

hth

[ograycode]

We were caught off guard by this, as cedar-14 is supported by Heroku until April of next year. We solved it both on cedar-14 and heroku-16 by removing this buildpack and instead using the apt buildpack provided by heroku for our django applications.

heroku-16

Aptfile:

libgeos-3.4.2
proj-bin
proj-data
libproj0
libgdal1h
gdal-bin
libblas3
liblapack3
libjasper1
libquadmath0

Commands:

heroku buildpacks:remove https://github.com/cyberdelia/heroku-geo-buildpack
heroku config:set GDAL_LIBRARY_PATH=/app/.apt/usr/lib/libgdal.so.1
heroku config:set LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib:/app/.apt/usr/lib/x86_64-linux-gnu:/app/.apt/usr/lib/i386-linux-gnu:/app/.apt/usr/lib:/app/.apt/usr/lib/libblas:/app/.apt/usr/lib/lapack
heroku config:set GEOS_LIBRARY_PATH=/app/.apt/usr/lib/x86_64-linux-gnu/libgeos_c.so.1
heroku stack:set heroku-16

settings.py:

GDAL_LIBRARY_PATH = os.environ.get('GDAL_LIBRARY_PATH', None)
GEOS_LIBRARY_PATH = os.environ.get('GEOS_LIBRARY_PATH', None)

cedar-14

Aptfile:

:repo:deb http://ppa.launchpad.net/ne3/trusty/ubuntu trusty main
:repo:deb http://ppa.launchpad.net/systemed/backportsvivid/ubuntu trusty main
libgeos-3.4.2
proj-bin
proj-data
libproj0
libgdal1h
gdal-bin
libblas3
liblapack3

commands:

heroku buildpacks:remove https://github.com/cyberdelia/heroku-geo-buildpack
heroku config:set GDAL_LIBRARY_PATH=/app/.apt/usr/lib/libgdal.so.1 
heroku config:set LD_LIBRARY_PATH=/app/.heroku/vendor/lib:/app/.heroku/python/lib:/app/.apt/usr/lib/x86_64-linux-gnu:/app/.apt/usr/lib/i386-linux-gnu:/app/.apt/usr/lib:/app/.apt/usr/lib/libblas:/app/.apt/usr/lib/lapack
heroku config:set GEOS_LIBRARY_PATH=/app/.apt/usr/lib/libgeos_c.so.1

settings.py:

GDAL_LIBRARY_PATH = os.environ.get('GDAL_LIBRARY_PATH', None)
GEOS_LIBRARY_PATH = os.environ.get('GEOS_LIBRARY_PATH', None)

notes

• This buildpack did not work on heroku-16 in a python environment using the instructions provided.

[ardell]

After trying and failing to get geos and proj to work on the heroku-16 stack with this buildpack I ended up switching to this one: https://github.com/diowa/heroku-buildpack-rgeo-prep

I ended up biting the bullet and upgrading to heroku-16 before deploying the new buildpack. I can't confirm that it works with cedar-14, but it seems like it might.

[volkanunsal]

Our build was broken again today. The new bucket seems to be inaccessible today. Is anyone else having this issue?

Edit: I just realized that the buildpack was added after the Ruby buildpack. I hope that resolves the problem.

[yc-yvonne]

@ograycode I'm trying your suggestion with the apt buildpack, and installation works but the GDAL_LIBRARY_PATH seems improperly configured for me (non existing file or directory error). How did you find the appropriate path to each of the dependencies?

[yc-yvonne]

For anyone coming on here after Oct 6, 2020, I ended up solving this issue and also issues with using the apt buildpack by using this one newly built by heroku with no additional config or setup: https://github.com/heroku/heroku-geo-buildpack

The reason adding lines like os.environ.get('GDAL_LIBRARY_PATH', None) to set the environment variable worked in the past is because they used to be set in the heroku/python buildpack, but that was an undocumented feature that was deprecated Oct 6: https://help.heroku.com/D5INLB1A/python-s-build_with_geo_libraries-legacy-feature-is-now-deprecated

For reference, this is how heroku/python used to set those variables: heroku/heroku-buildpack-python@f767a73