Replace opensource.apple.com/tarballs to macports_distfiles #18956

catap · 2023-06-03T11:15:49Z

Description

Apple moved all sources from opensource.apple.com/tarballs to github, that leads to a different distfiles which hadn't passed checksum.

Unfortunately they hadn't migrated everything, for example some patches for ld64 are missed.

But, we have it inside distfiles => let use distfiles as mirror.

Type(s)

bugfix

Tested on

macOS 10.5.8 9L34 i386
Xcode 3.1.4 9M2809

Verification

Have you

followed our Commit Message Guidelines?
squashed and minimized your commits?
checked that there aren't other open pull requests for the same change?
checked your Portfile with port checksum?

macportsbot · 2023-06-03T11:15:54Z

Notifying maintainers:
@jeremyhu for port ld64, libmacho, llvm-gcc42.
@raimue for port bsdmake.

kencu · 2023-06-03T14:55:26Z

some ports are static and will never be changed, so doesn't really matter for them.

but if you do this for ld64, sometime we'll need to undo it again when it gets updated, eg

kencu@5358f2d

catap · 2023-06-03T14:58:56Z

@kencu or tag this site as macports and use it for old ld64.

Anyway, right now you aren't able to build ld64 on 10.5 because it hasn't got binary distirbutives and dowloads from github an archive which doesn't match.

Thus, apple not just change archive checksum, it changes it and it requires some work to fix.

catap · 2023-06-03T15:03:41Z

@kencu anyway, may I ask you to merge it? :) At least it covers an issue that exists right now.

kencu · 2023-06-03T15:15:14Z

I'll leave this for @jmroot -- it's a big change, and we'll let the admins own it.

raimue · 2023-06-04T16:17:07Z

It is unlikely that any upcoming version will be placed on opensource.apple.com, but they only seem to use GitHub now. Wouldn't it make sense to switch these ports to always use downloads from GitHub and update the checksums (with a revision bump)? I guess the different checksum is likely due to the subdirectory name introduced by GitHub, but the contents should be the same.

catap · 2023-06-04T23:13:52Z

@raimue unfortunately some files are missed. For example patch which is refered from ld64 ports. Also, they had repack a bit sources and some work were required to migrate to github sources for cctools: a079d1b

Right now I have an issue: I can't install any of this ports because it downloads archive from github and checksum isn't match.

cooljeanius · 2023-10-28T14:55:49Z

Right now I have an issue: I can't install any of this ports because it downloads archive from github and checksum isn't match.

Running into this issue with a few of the ld64 ports now, too... for ld64-236, the error is:

DEBUG: checksum phase started at Sat Oct 28 10:48:05 EDT 2023
--->  Verifying checksums for ld64-236
DEBUG: Executing org.macports.checksum (ld64-236)
--->  Checksumming dyld-655.1.1.tar.gz
DEBUG: Calculated (rmd160) is 3ce28be9a3eb52777443828853040d4c870a5db2
Error: Checksum (rmd160) mismatch for dyld-655.1.1.tar.gz
Portfile checksum: dyld-655.1.1.tar.gz rmd160 eef3ce30381e5fa23152c637e3bde2d8cf4cde0b
Distfile checksum: dyld-655.1.1.tar.gz rmd160 3ce28be9a3eb52777443828853040d4c870a5db2
DEBUG: Calculated (sha256) is 67326487dd139a82f0da9f43eb91414041551ee54cacf4763052e5aae606f518
Error: Checksum (sha256) mismatch for dyld-655.1.1.tar.gz
Portfile checksum: dyld-655.1.1.tar.gz sha256 8ca6e3cf0263d3f69dfa65e0846e2bed051b0cff92e796352ad178e7e4c92f1d
Distfile checksum: dyld-655.1.1.tar.gz sha256 67326487dd139a82f0da9f43eb91414041551ee54cacf4763052e5aae606f518
DEBUG: Calculated (size) is 872742
Error: Checksum (size) mismatch for dyld-655.1.1.tar.gz
Portfile checksum: dyld-655.1.1.tar.gz size 907171
Distfile checksum: dyld-655.1.1.tar.gz size 872742
--->  Checksumming ld64-236.3.tar.gz
DEBUG: Calculated (rmd160) is 1811d7bf9a72949879716976b06c9b52455da990
Error: Checksum (rmd160) mismatch for ld64-236.3.tar.gz
Portfile checksum: ld64-236.3.tar.gz rmd160 6a3f44aa9ae57a60d2cff5b3d47be7972ad83029
Distfile checksum: ld64-236.3.tar.gz rmd160 1811d7bf9a72949879716976b06c9b52455da990
DEBUG: Calculated (sha256) is d88b3bf58fc5cfe7c33babfc9619dbb82a5a4a0b4a5f44aa12b60b6efe642756
Error: Checksum (sha256) mismatch for ld64-236.3.tar.gz
Portfile checksum: ld64-236.3.tar.gz sha256 8ef36729b643201081ab45ebd8586ede8f9968bc17614b679a940faa82875ca6
Distfile checksum: ld64-236.3.tar.gz sha256 d88b3bf58fc5cfe7c33babfc9619dbb82a5a4a0b4a5f44aa12b60b6efe642756
DEBUG: Calculated (size) is 604369
Error: Checksum (size) mismatch for ld64-236.3.tar.gz
Portfile checksum: ld64-236.3.tar.gz size 624400
Distfile checksum: ld64-236.3.tar.gz size 604369
The correct checksum line may be:
checksums           dyld-655.1.1.tar.gz \
                    rmd160  3ce28be9a3eb52777443828853040d4c870a5db2 \
                    sha256  67326487dd139a82f0da9f43eb91414041551ee54cacf4763052e5aae606f518 \
                    size    872742 \
                    ld64-236.3.tar.gz \
                    rmd160  1811d7bf9a72949879716976b06c9b52455da990 \
                    sha256  d88b3bf58fc5cfe7c33babfc9619dbb82a5a4a0b4a5f44aa12b60b6efe642756 \
                    size    604369
Error: Failed to checksum ld64-236: Unable to verify file checksums
DEBUG: Error code: NONE
DEBUG: Backtrace: Unable to verify file checksums

...and for ld64-274 it is:

DEBUG: checksum phase started at Sat Oct 28 10:54:38 EDT 2023
--->  Verifying checksums for ld64-274
DEBUG: Executing org.macports.checksum (ld64-274)
--->  Checksumming dyld-655.1.1.tar.gz
DEBUG: Calculated (rmd160) is 3ce28be9a3eb52777443828853040d4c870a5db2
Error: Checksum (rmd160) mismatch for dyld-655.1.1.tar.gz
Portfile checksum: dyld-655.1.1.tar.gz rmd160 eef3ce30381e5fa23152c637e3bde2d8cf4cde0b
Distfile checksum: dyld-655.1.1.tar.gz rmd160 3ce28be9a3eb52777443828853040d4c870a5db2
DEBUG: Calculated (sha256) is 67326487dd139a82f0da9f43eb91414041551ee54cacf4763052e5aae606f518
Error: Checksum (sha256) mismatch for dyld-655.1.1.tar.gz
Portfile checksum: dyld-655.1.1.tar.gz sha256 8ca6e3cf0263d3f69dfa65e0846e2bed051b0cff92e796352ad178e7e4c92f1d
Distfile checksum: dyld-655.1.1.tar.gz sha256 67326487dd139a82f0da9f43eb91414041551ee54cacf4763052e5aae606f518
DEBUG: Calculated (size) is 872742
Error: Checksum (size) mismatch for dyld-655.1.1.tar.gz
Portfile checksum: dyld-655.1.1.tar.gz size 907171
Distfile checksum: dyld-655.1.1.tar.gz size 872742
--->  Checksumming ld64-274.2.tar.gz
DEBUG: Calculated (rmd160) is 1dc9dd59ad2c760e2920a40890f9ca614f20cc16
Error: Checksum (rmd160) mismatch for ld64-274.2.tar.gz
Portfile checksum: ld64-274.2.tar.gz rmd160 16e57faf2b1d3f3808fb8e4b167cf23a14f106b0
Distfile checksum: ld64-274.2.tar.gz rmd160 1dc9dd59ad2c760e2920a40890f9ca614f20cc16
DEBUG: Calculated (sha256) is 89f5fe683faff0da8990cf881beaa0ca61bd13827d4a75aada423a8693a68714
Error: Checksum (sha256) mismatch for ld64-274.2.tar.gz
Portfile checksum: ld64-274.2.tar.gz sha256 175d89c419e99d49a7a5f7e4196d3cef4c9e19cc17a425c332e86df6b516f7d7
Distfile checksum: ld64-274.2.tar.gz sha256 89f5fe683faff0da8990cf881beaa0ca61bd13827d4a75aada423a8693a68714
DEBUG: Calculated (size) is 674878
Error: Checksum (size) mismatch for ld64-274.2.tar.gz
Portfile checksum: ld64-274.2.tar.gz size 671652
Distfile checksum: ld64-274.2.tar.gz size 674878
The correct checksum line may be:
checksums           dyld-655.1.1.tar.gz \
                    rmd160  3ce28be9a3eb52777443828853040d4c870a5db2 \
                    sha256  67326487dd139a82f0da9f43eb91414041551ee54cacf4763052e5aae606f518 \
                    size    872742 \
                    ld64-274.2.tar.gz \
                    rmd160  1dc9dd59ad2c760e2920a40890f9ca614f20cc16 \
                    sha256  89f5fe683faff0da8990cf881beaa0ca61bd13827d4a75aada423a8693a68714 \
                    size    674878
Error: Failed to checksum ld64-274: Unable to verify file checksums

...and for ld64-latest it is:

DEBUG: checksum phase started at Sat Oct 28 10:55:26 EDT 2023
--->  Verifying checksums for ld64-latest
DEBUG: Executing org.macports.checksum (ld64-latest)
--->  Checksumming dyld-655.1.1.tar.gz
DEBUG: Calculated (rmd160) is 3ce28be9a3eb52777443828853040d4c870a5db2
Error: Checksum (rmd160) mismatch for dyld-655.1.1.tar.gz
Portfile checksum: dyld-655.1.1.tar.gz rmd160 eef3ce30381e5fa23152c637e3bde2d8cf4cde0b
Distfile checksum: dyld-655.1.1.tar.gz rmd160 3ce28be9a3eb52777443828853040d4c870a5db2
DEBUG: Calculated (sha256) is 67326487dd139a82f0da9f43eb91414041551ee54cacf4763052e5aae606f518
Error: Checksum (sha256) mismatch for dyld-655.1.1.tar.gz
Portfile checksum: dyld-655.1.1.tar.gz sha256 8ca6e3cf0263d3f69dfa65e0846e2bed051b0cff92e796352ad178e7e4c92f1d
Distfile checksum: dyld-655.1.1.tar.gz sha256 67326487dd139a82f0da9f43eb91414041551ee54cacf4763052e5aae606f518
DEBUG: Calculated (size) is 872742
Error: Checksum (size) mismatch for dyld-655.1.1.tar.gz
Portfile checksum: dyld-655.1.1.tar.gz size 907171
Distfile checksum: dyld-655.1.1.tar.gz size 872742
--->  Checksumming ld64-450.3.tar.gz
DEBUG: Calculated (rmd160) is ffbf7b2dcd9826db463c779c6ae920f4a71803d3
Error: Checksum (rmd160) mismatch for ld64-450.3.tar.gz
Portfile checksum: ld64-450.3.tar.gz rmd160 4d263cff143228e021c438d3c2d1800ff367c895
Distfile checksum: ld64-450.3.tar.gz rmd160 ffbf7b2dcd9826db463c779c6ae920f4a71803d3
DEBUG: Calculated (sha256) is 7c14d80f1552819e7038e918dde9cc7b2398e08678471221330f173c87cf565a
Error: Checksum (sha256) mismatch for ld64-450.3.tar.gz
Portfile checksum: ld64-450.3.tar.gz sha256 140619e676e099581771dbad98277850ff731cd23938bed95b4d7171616acca1
Distfile checksum: ld64-450.3.tar.gz sha256 7c14d80f1552819e7038e918dde9cc7b2398e08678471221330f173c87cf565a
DEBUG: Calculated (size) is 716654
Error: Checksum (size) mismatch for ld64-450.3.tar.gz
Portfile checksum: ld64-450.3.tar.gz size 729639
Distfile checksum: ld64-450.3.tar.gz size 716654
The correct checksum line may be:
checksums           dyld-655.1.1.tar.gz \
                    rmd160  3ce28be9a3eb52777443828853040d4c870a5db2 \
                    sha256  67326487dd139a82f0da9f43eb91414041551ee54cacf4763052e5aae606f518 \
                    size    872742 \
                    ld64-450.3.tar.gz \
                    rmd160  ffbf7b2dcd9826db463c779c6ae920f4a71803d3 \
                    sha256  7c14d80f1552819e7038e918dde9cc7b2398e08678471221330f173c87cf565a \
                    size    716654
Error: Failed to checksum ld64-latest: Unable to verify file checksums
DEBUG: Error code: NONE
DEBUG: Backtrace: Unable to verify file checksums

catap · 2023-11-13T12:06:47Z

Here we go again. I've switched to fetching distfiles from github for all ports with two exceptions: appel-gcc40 and apple-gcc42; do we really need it?

kencu · 2023-11-13T16:32:13Z

apple-gcc42 is required for Tiger to bootstrap and function.

catap · 2023-11-13T17:58:32Z

Ken, here it is. Anyway, I can't switch apple-gcc40 to GitHub and keep it switched to MacPorts
distfiles, because GitHub contains only one tag 4064 when port is used 5494 which seems to be missed.

curl -D - -L http://opensource.apple.com/tarballs/gcc_40/gcc_40-5494.tar.gz returns 404 on my end.

kencu · 2023-11-13T19:42:40Z

there are a bunch here too

https://github.com/apple-oss-distributions/gcc/tags

catap · 2023-11-13T20:47:39Z

Ken, good idea.

It had missed 5494 build, but has 5493 only. So, I've created a trivial patch to keep the sources as it was before.

jmroot · 2023-11-14T01:27:03Z

devel/CoreOSMakefiles/Portfile

@@ -2,6 +2,7 @@ PortSystem              1.0

 name                    CoreOSMakefiles
 version                 77
+revision                1


Don't bump the revision if the code isn't changing.

Looks like that's fixed as of 6d86c28.

Yes, I did. But I haven't click "resolve conversation" to make things clear and allow Joshua to see that his remarks was addressed.

jmroot · 2023-11-14T01:29:59Z

devel/CoreOSMakefiles/Portfile

+                        sha256  b8bf4c3fa727d71615665322bb22813eda15962544df5238cc37aa606bda276c \
+                        size    17200
+
+dist_subdir             apple-oss-distributions


Why would we want all these ports to share a dist_subdir?

6d86c28 has instead:

# simulate stealth update; remove with next version change dist_subdir ${name}/${version}_1

sames here.

jmroot · 2023-11-14T01:31:01Z

devel/CoreOSMakefiles/Portfile

+                        size    17200
+
+dist_subdir             apple-oss-distributions
+extract.rename          yes


The worksrcdir doesn't have anything unpredictable like a hash in it, so prefer to set that correctly rather than set extract.rename.

Nope, it extracts like CoreOSMakefiles-CoreOSMakefiles-77, and the same pattern is applied for all that ports.

jmroot · 2023-11-14T01:31:40Z

devel/bootstrap_cmds/Portfile

@@ -3,20 +3,23 @@ PortGroup               xcode 1.0

 name                    bootstrap_cmds
 version                 121.100.1
-epoch                   1
+epoch                   2


There's no reason to change the epoch since the version isn't changing.

it was a typo, I plan to increase revision.

mark as resolved, then?

edit: confirmed that the epoch bump is gone in 6d86c28

yep, it is resolved.

jmroot · 2023-11-14T01:33:45Z

lang/apple-gcc40/Portfile

+distfiles       gcc-5493${extract.suffix}
+# Apple's github had missed 5494 build but it had only 5493,
+# here recovery minimal differences betwen them
+patchfiles      apple-gcc-5493-5494.diff


If this one is missing upstream, it's probably fine to just use macports_distfiles for the master_sites.

@jmroot but that patch is extremely small; I'd like to say almost nothing.

yeah, the patch is only 51 lines, of which only 2 are functional (3 if you include the version number bump)

catap · 2023-11-21T21:50:43Z

@pmetzger can it be merged?

cooljeanius

Looks mostly fine to me, although I'm wondering about the ld64 lint warnings. Note that I haven't actually tested myself, just looked things over. Will approve once question about ld64 lint warnings is answered (if they were already there beforehand, then they're fine to ignore, I guess, but if they're new, I'd think they should be addressed)

devel/ld64/Portfile

Apple moved all sources from opensource.apple.com/tarballs to GitHub, that leads to a different distfiles which hadn't passed checksum. Closes: https://trac.macports.org/ticket/68538

macportsbot assigned jeremyhu and raimue Jun 3, 2023

macportsbot added type: bugfix maintainer: requires approval labels Jun 3, 2023

catap marked this pull request as draft June 7, 2023 13:09

catap mentioned this pull request Aug 16, 2023

unifdef: add conflict to developer_cmds #19918

Merged

5 tasks

cooljeanius mentioned this pull request Nov 11, 2023

dumped emacs is unusable cooljeanius/emacs#1

Open

catap force-pushed the ld64 branch 2 times, most recently from f5b1c66 to f3c79bd Compare November 13, 2023 12:05

catap marked this pull request as ready for review November 13, 2023 12:05

catap force-pushed the ld64 branch from f3c79bd to eb54d3c Compare November 13, 2023 17:57

catap force-pushed the ld64 branch from eb54d3c to 36f1f31 Compare November 13, 2023 20:47

jmroot requested changes Nov 14, 2023

View reviewed changes

catap force-pushed the ld64 branch from 36f1f31 to 6d86c28 Compare November 14, 2023 20:51

catap requested a review from jmroot November 14, 2023 20:53

catap requested a review from cooljeanius November 21, 2023 21:50

cooljeanius suggested changes Nov 21, 2023

View reviewed changes

devel/ld64/Portfile Outdated Show resolved Hide resolved

cooljeanius approved these changes Nov 21, 2023

View reviewed changes

jmroot force-pushed the ld64 branch from 6d86c28 to 7ec8666 Compare November 22, 2023 00:06

Migrate opensource.apple.com/tarballs to GitHub

4c6f9b4

Apple moved all sources from opensource.apple.com/tarballs to GitHub, that leads to a different distfiles which hadn't passed checksum. Closes: https://trac.macports.org/ticket/68538

jmroot force-pushed the ld64 branch from 7ec8666 to 4c6f9b4 Compare November 22, 2023 00:11

jmroot approved these changes Nov 22, 2023

View reviewed changes

jmroot merged commit a55cc41 into macports:master Nov 22, 2023
3 checks passed

catap deleted the ld64 branch November 22, 2023 09:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace opensource.apple.com/tarballs to macports_distfiles #18956

Replace opensource.apple.com/tarballs to macports_distfiles #18956

catap commented Jun 3, 2023

macportsbot commented Jun 3, 2023

kencu commented Jun 3, 2023

catap commented Jun 3, 2023

catap commented Jun 3, 2023

kencu commented Jun 3, 2023

raimue commented Jun 4, 2023

catap commented Jun 4, 2023

cooljeanius commented Oct 28, 2023

catap commented Nov 13, 2023

kencu commented Nov 13, 2023

catap commented Nov 13, 2023 •

edited

Loading

kencu commented Nov 13, 2023

catap commented Nov 13, 2023

jmroot Nov 14, 2023

cooljeanius Nov 20, 2023

catap Nov 20, 2023

jmroot Nov 14, 2023

cooljeanius Nov 20, 2023 •

edited

Loading

catap Nov 20, 2023

jmroot Nov 14, 2023

catap Nov 14, 2023

jmroot Nov 14, 2023

catap Nov 14, 2023

cooljeanius Nov 20, 2023 •

edited

Loading

catap Nov 20, 2023

jmroot Nov 14, 2023

catap Nov 14, 2023

cooljeanius Nov 20, 2023

catap commented Nov 21, 2023

cooljeanius left a comment

Replace opensource.apple.com/tarballs to macports_distfiles #18956

Replace opensource.apple.com/tarballs to macports_distfiles #18956

Conversation

catap commented Jun 3, 2023

Description

Type(s)

Tested on

Verification

macportsbot commented Jun 3, 2023

kencu commented Jun 3, 2023

catap commented Jun 3, 2023

catap commented Jun 3, 2023

kencu commented Jun 3, 2023

raimue commented Jun 4, 2023

catap commented Jun 4, 2023

cooljeanius commented Oct 28, 2023

catap commented Nov 13, 2023

kencu commented Nov 13, 2023

catap commented Nov 13, 2023 • edited Loading

kencu commented Nov 13, 2023

catap commented Nov 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cooljeanius Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cooljeanius Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

catap commented Nov 21, 2023

cooljeanius left a comment

Choose a reason for hiding this comment

catap commented Nov 13, 2023 •

edited

Loading

cooljeanius Nov 20, 2023 •

edited

Loading

cooljeanius Nov 20, 2023 •

edited

Loading