Travis CI OS X Platform support/OS X Compatibility

[jakirkham]

Travis CI has switched from OS X 10.9 with XCode 6.1 to 10.11 with XCode 7.3 as their default OS for Mac. It appears this was announced on their blog and was switched out yesterday. However, we weren't aware of this change and so have not done anything about this until now. As a consequence, there is ~1days worth of packages that may not be compatible with 10.9 (more investigation will be required) and need to be rebuilt to restore that compatibility.

To switch back to the previously used base image, fixes have been put in play in staged-recipes with PR ( conda-forge/staged-recipes#1717 ) and conda-smithy with PR ( conda-forge/conda-smithy#325 ) releases as part of v1.3.2. Feedstocks created by staged-recipes from this point forward should be on the supported platform 10.9. Any feedstock that has not been re-rendered should be re-rendered with 1.3.2 before doing any build number bumps or new releases (though this can be part of the same PR).

It appears that Travis CI plans on dropping 10.9 at the end of the month. So the oldest OS we can work with will be 10.10. If we wish to still support 10.9, we should do one or more of the following.

1. Raise an issue upstream with Travis CI requesting the image remain.
2. Scope out other CIs that will meet or allow us to meet this requirement.
3. Try to install the 10.9 SDK in the VMs ourselves and build against it.
   • https://github.com/devernay/xcodelegacy
   • https://github.com/phracker/MacOSX-SDKs
4. Bump our OS X requirements to 10.10.

If you know of options meeting these or some other option, please raise them. Please share your thoughts. Will be raising this in our next meeting.

Thanks to @dopplershift who raised a lot of this information to our attention in PR ( conda-forge/staged-recipes#1716 ).

Edit (pulled in from comment): Raised this on gitter as well.

Edit: Have raised issues or made comments on open PRs to encourage re-renderings on feedstocks that compiled code on 10.11 recently that need to be moved back to 10.9 and rebuilt.

cc @conda-forge/core

[jakirkham]

Regardless of what we decide, it would be really healthy if we could start out by doing a survey of our users' needs.

[dopplershift]

Isn't it possible to specify a build target of 10.9 for Xcode or whatnot? It's not like Apple has dropped support for 10.9, so other places are building for it without having 10.9 build machines.

[jakirkham]

That's not so simple. One needs to have the SDK for that OS version in their copy of XCode. Often XCode does drop old SDKs.

Would need to double check if 10.9 is included in XCode 6.4, but remember there was some uncertainty when we last looked at this. Also has been awhile since I last looked into this. Please see PR ( conda-forge/staged-recipes#1094 ). Particularly this comment. Maybe @jjhelmus can share more of what he found.

It is possible to install our own copy of the excluded SDK into XCode and use it (option 3). Though I guess that gets into some murky waters as far as support is concerned. Others have done this successfully. IIRC @patricksnape has some experience with this.

---

On a different note, would you be able to come to our meeting @dopplershift ? Would be nice to have another Mac voice around. The time is YTBD, but could be good if you wish to start coming regularly. Please see this poll for options. If you don't wish to vote/want to commit to more meetings, but would be interested in being at this meeting, can try to give you an update on the time once that is settled.

[jjhelmus]

It is possible to build binaries which target an older versions of OS X using a base SDK from a later version (i.e. target 10.9 using an 10.10 SDK). Apple's SDK Compatibility Guide provides a nice explanation.

The mechanism for this is to set the MACOSX_DEPLOYMENT_TARGET environment variable to the desired target version. Alternatively the --mmacosx-version-min flag can be passed to gcc (and maybe clang?). These two mechanisms seem to be synonymous.

[jakirkham]

Guessing you would be very interested in this discussion, @183amir, given the number of compiled packages you have on OS X.

[dopplershift]

I may be able to make it tomorrow (it's 8am local, so have to squeeze around getting everyone out of the house).

[jakirkham]

That's great! Have added this topic to the agenda.

[jakirkham]

Alright @conda-forge/core , it is 14 October now we are at about the halfway point of the month. It would be really great if everyone could comment on this issue particularly about this point.

[dopplershift]

Is the expected challenge having the package built on 10.10, but work on 10.9? That might limit my testing, since I don't know where I may find a 10.9 box. I can test on 10.12, but I doubt that's really going to matter.

[jakirkham]

Yes, I have 10.9 to test on. Though I don't have the bandwidth to build lots of stuff. So if you have time to build some things with that Travis VM maybe outside conda-forge, I can try them out.

[wesm]

I'm running into problems with XCode 6.1 that seem to go away with XCode 6.4 (related to rpath / @loader_path, see conda-forge/staged-recipes#1732) -- this would be a huge help, otherwise I won't be able to build OS X packages for this project until the architecture is upgraded. Let me know if I can help -- unfortunately, my only mac is now on XCode 8.0 and there are no issues there.

[jakirkham]

You're in good company @wesm . PR ( conda-forge/staged-recipes#1481 ) which adds a recipe for clang needed for things like numba is in the same boat.

[jakirkham]

Am asking upstream to hold onto the existing image, beta-xcode6.1, a little longer as I don't think we are going to make the cutoff, which is the end of this month. Please see issue ( travis-ci/travis-ci#6765 ). I still do think we should switch to xcode6.4, but I think we need to do some retooling to do this more effectively.

Sending gobs of re-rendering PRs out is at best slow and inefficient given the number of feedstocks and how many of them are actually compiling something. In the worst case, it can throttle CIs that may be already struggling. Paying attention to active feedstocks instead could give us the needed savings and be a bit more effective. ( conda-forge/conda-forge-webservices#74 ) Also re-rendering in all pinning PRs should avoid some common mistakes. ( conda-forge/conda-forge-maintenance#12 )

[mwcraig]

It sounds like we won't have much choice about moving to the newer version of xcode, judging by the initial reaction to the issue you opened.

How many packages actually need to be rebuilt (I know the travis configuration file will change) with this change?

[mwcraig]

One more (maybe irrelevant?) data point: I had a build which fails (wasn't using toolchain to set the compile options correctly) under XCode 7.3 but builds successfully under 6.1. Only change was XCode version/travis image. Trying it on 6.4 in a little bit...

[jakirkham]

One more (maybe irrelevant?) data point: I had a build which fails (wasn't using toolchain to set the compile options correctly) under XCode 7.3 but builds successfully under 6.1. Only change was XCode version/travis image. Trying it on 6.4 in a little bit...

Yeah, so there may have been some changes to install_name_tool or default configurations between XCode versions.

In any event, the issue you experienced comes down to not having enough room in the dylib header would be easily solved by using the toolchain as it automatically sets -headerpad_max_install_names during the build. This ensures the header is filled to its maximum length (mostly likely with NULL characters), which ensures it will be easier to change later.

While not necessarily an issue with XCode 6.4, it is worth noting that there may be a variety of issues that may come from upgrading to the new image. Some may be easily solved by switching to the toolchain. Others may be more involved. It certainly wouldn't hurt for some people to try out switching images in a PR on their feedstocks or other places to see how this goes.

This all being said, the previous deadline set by Travis CI was 2016 Oct 31. So we really don't have much time to experiment. It may just be that the upgrade itself will be the experiment.

[jakirkham]

It sounds like we won't have much choice about moving to the newer version of xcode, judging by the initial reaction to the issue you opened.

You're probably right. However, it seems some core members including myself have been swamped with day jobs/non-conda-forge activities at this time. So despite the urgency of this issue, we have been unable to sufficiently address it. Thus I thought it might be worthwhile to beg for more time. 😄

How many packages actually need to be rebuilt (I know the travis configuration file will change) with this change?

So this is actually a piece of good news, nothing should actually need to be rebuilt. Also the other piece of good news is the xcode6.4 image contains the 10.9 SDK. So we don't need to worry about compatibility changes on that front (assuming we are setting our base version of compatibility correctly).

However, problems will occur when people try to make new releases or build number bumps.

If the image is not specified (as your experience elucidates), Travis CI puts us on the default image, which is 10.11 with XCode 7.3. That version of XCode FWICT does not contain 10.9 or 10.10 SDKs only 10.11. Also even if they are using the toolchain or setting MACOSX_DEPLOYMENT_TARGET explicitly, we have found this may still not be sufficient for selecting the SDK as explained in this comment.

Even if the image is specified, but doesn't exist, Travis CI seems to quietly change back to the default image ( travis-ci/travis-ci#6684 ). As a result, once beta-xcode6.1 is dropped people will quietly be bumped to xcode7.3 even though that is clearly not their intention. At a bare minimum, getting a build failure in these cases would provide some impetus to re-render with conda-smithy.

Though there is still a final lurking concern (briefly alluded to). Even if we agree that we want to use xcode6.4, and configure conda-smithy and staged-recipes to use this and we do push all new recipes to use the toolchain to set the Mac OS deployment target to 10.9 or at least set the MACOSX_DEPLOYMENT_TARGET, how do we ensure maintainers of existing feedstocks are aware of this problem and address it?

Right now the best mechanism we have is re-rendering PRs. Though as we have seen of late that mechanism is far from sufficient. Solving this problem ends up being the tricky one. It probably will be a mixture of making conda-smithy super easy to use, providing automated re-rendering PRs primarily to actively used feedstocks quickly, and adding various CI level checks to ensure compatibility by failing the build if not met. Doing all of this will take time of several people at least and it's doubtful this will happen in a week. Hence the request to Travis.

Maybe one thing that can help consolidate this problem a little more with our other looming problem conda-build version 2.* support is to make MACOSX_DEPLOYMENT_TARGET a whitelisted variable in conda-build and set this in conda-forge-build-setup. Thus to get that to work we will need to switch to a 2.* release that has that fix. Perhaps by doing this, we will have a better chance of making a good effort on both.

[183amir]

Maybe toolchain can throw an error during activation when the 10.9 SDK is not available?

[jakirkham]

Not everything has the toolchain unfortunately. However I think you have the right idea and we certainly could. Maybe we could also add a similar check to conda-forge-build-setup. Still doesn't catch everything (like really old feedstocks), but it should already be very helpful.

[183amir]

Eventually, we should make sure that all packages which compile stuff (or maybe all all) should have toolchain in their build requirements. I already did that for all bob packages today it was just a simple regular expression script.

[mwcraig]

Thus I thought it might be worthwhile to beg for more time

@jakirkham -- definitely, thanks for doing it! I've been swamped with non-conda stuff for a while and haven't had basically any time to spend on conda-forge (or other building repos) since late July.

I have built a couple things on travis osx with a couple combinations of image/xcode version all targeting 10.9. I could upload those if you could test on your box.

In any event, it sounds like we are in a bad place if the 6.1 image goes away Monday. If there is anything I can do to help, let me know.

[jakirkham]

So just heard back from Travis CI. It sounds like we will continue to have the 6.1 image for a few more months. However, they are not planning to provide support for it, which sounds totally reasonable. In other words, we have some more time to work on upgrading, but we still need to be working towards that goal.

[mwcraig]

@jakirkham -- thanks again for making the request!

[jakirkham]

cc @inducer

[jakirkham]

I'd like to make a request of the community. If someone has the bandwidth and interest to upgrade the OS X image on Travis CI, please put together an enhancement proposal. This has been request by a few of the core members. While I'm interested in this change, I sadly find I don't nearly have enough time to do this at present. It looks like there are several use cases where XCode 6.4 solves problems and soon we will need to move to it anyways. Personally I'm an advocate of such a move and would be happy to be listed as a champion of such a proposal/provide general guidance through the process. Thanks in advance.

[inducer]

• To play devil's advocate, what's the advantage of the CFEP process for this decision?
• What all goes into it, techncially, other than flipping the image name? Is a package rebuild required?

[jakirkham]

So the point of the CFEP is to do exactly that. Outline what actions need to be taken to switch VM, compiler, etc. and what the implications are. It is also there to compare other options and explain why the proposed option (in the most objective way possible) is the best course of action.

Beyond these technical merits the CFEP system is in place to ensure greater transparency w.r.t. org wide changes. While it is true that everything here is done in the open, changing the VM image in a PR alone at staged-recipes can easily be lost in the noise. This is bad both for the person proposing, interested third parties, and core. Namely because it will not get the attention it deserves. Putting together a CFEP will make it clear this change is proposed and its implications in front of the intended audience without unneeded distractions.

Hopefully that gives you a better idea of the value. 😉

[inducer]

Here's a very quick straw man for that CFEP. Could use fact check/edit/suggestions.

[matthew-brett]

I got the same response as you when emailing travis-ci support. Homebrew dropping support, try using 10.9 sdk from 10.10 image.

[jakirkham]

At this point, staged-recipes is on 10.10 and any feedstock re-rendered with conda-smithy 2.0.0 is on 10.10. New feedstocks are starting out with 10.10 automatically. While more feedstocks need to be re-rendered to be switched to the image, that seems to be happening organically. Given all of this, am closing this issue out.