BUILDPACK_VENDOR_URL doesn't allow for using vendored Ruby since v201

[dannyfallon]

A long time ago we forked this Ruby buildpack and recently I've been looking at going back to using upstream. One of the changes I've noticed is that prior to v227 it was possible to specify BUILDPACK_VENDOR_URL and this is where Ruby binaries would be fetched from. Since the extraction of the Ruby binary downloading out into bash in specifically #886 the BUILDPACK_VENDOR_URL env var is ignored and binaries are always fetched from https://s3-external-1.amazonaws.com/heroku-buildpack-ruby.

This feature to allow getting binaries from different locations was initially added way back in #238 and has proliferated into other Heroku buildpacks since then e.g. Python used it, but recently swapped to use BUILDPACK_S3_BASE_URL in heroku/heroku-buildpack-python#1085). The lack of support for it now for Ruby binaries seems at odds with the fact VENDOR_URL is still being used elsewhere in the buildpack today (e.g. plugins)

Is this something that you'd be interested in adding support for with the new downloader system? Without that feature we're going to be forced from the off to keep our fork, regardless of anything else that might crop up 😅

[schneems]

Sorry for the delay. I didn't see this come in and then I went out of office for a bit. I think we could replace this string "https://s3-external-1.amazonaws.com/heroku-buildpack-ruby" in bash with:

heroku_buildpack_ruby_url=${BUILDPACK_VENDOR_URL:"https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/$STACK/ruby-${BASH_REMATCH[1]}.tgz"}

Or we could set BUILDPACK_VENDOR_URL when undefined and use that env var everywhere.

I am curious what functionality you're after by hosting your own binaries if this is a prod or a dev concern. Maybe there's another way to achieve what you're after with a more stable interface.

Ultimately we need to be able to point the buildpack at our staging binary bucket so we need this fix either way. Ideally we would also add some sort of an integration test to prevent another regression.

[dannyfallon]

👋 No worries on the delay. We're not on the Heroku platform but are running on AWS. This is a production concern for us on two fronts - security and performance.

From a security perspective we like to vendor and build our own Ruby binary to ensure binaries are available after a CVE reasonably quickly. Tangentially we also reduce the attack surface by keeping the dependency in-house, albeit to limited effect since we're not vendoring all our gems 😅

On performance, until 2020 we were building Ruby binaries with GCC's default Intel x86_64 architecture but investigation using GCC9 and targeting the Skylake architecture and turning for Cannonlake (CFLAGs -m64 -march=skylake -mtune=cannonlake) we achieved ~8.5% better performance in optcarrot. When we rolled that out in production we saw a not-insignificant reduction the number of machines running during peak. I've not enquired about how Heroku compiles the binaries for its platform, but I do think it's unlikely we'll end up with your build architecture targets matching ours.

I hope this makes sense and I'd be happy to give a solution a try in my free time if you've a preferred solution

[schneems]

we achieved ~8.5% better performance in optcarrot

Wow, that's awesome. I've not looked into any compile time optimizations. Here's our compilation repo https://github.com/heroku/docker-heroku-ruby-builder. You can see where we do the make here https://github.com/heroku/docker-heroku-ruby-builder/blob/f1b4ee77a8d891bf2f72ecca4c28ad2a3e96b6cb/build.rb#L167-L169.

I'd be happy to give a solution a try in my free time if you've a preferred solution

If you've got the time that would be great. I'm in the process of a re-write https://github.com/heroku/buildpacks-ruby also i'm working on supporting salesforce functions (serverless product we're rolling out under the name "evergreen") for node and java. This current repo is still getting time and attention, but my efforts are split right now.

I think the most consistent solution might be to check for the existance of BUILDPACK_VENDOR_URL and if it's not set, set it to https://s3-external-1.amazonaws.com/heroku-buildpack-ruby early on and then have the code use BUILDPACK_VENDOR_URL env var everywhere. I'm also up for discussion/exploration if that turns out to be complicated or difficult.

It's also worth thinking about the rewrite logic as well. I've not yet implemented this feature in the new repo. I'm wondering if it makes sense to have more granular over-rides (for instance, so I could use different ruby binaries but keep the same bundler path). I'm also thinking we should probably prefix this with a HEROKU_RUBY_* to avoid colission with other buildpacks though it would make the env var quite long.

[dzuelke]

On performance, until 2020 we were building Ruby binaries with GCC's default Intel x86_64 architecture but investigation using GCC9 and targeting the Skylake architecture and turning for Cannonlake (CFLAGs -m64 -march=skylake -mtune=cannonlake) we achieved ~8.5% better performance in optcarrot.

Heroku runs on instance types much older than Skylake/Cannonlake.