Skip to content

[vcpkg-acquire-msys] Update pacman before any other package.#11443

Merged
BillyONeal merged 2 commits intomicrosoft:masterfrom
emptyVoid:fix-issue-11438
May 22, 2020
Merged

[vcpkg-acquire-msys] Update pacman before any other package.#11443
BillyONeal merged 2 commits intomicrosoft:masterfrom
emptyVoid:fix-issue-11438

Conversation

@emptyVoid
Copy link
Contributor

Describe the pull request

Adjusts vcpkg_acquire_msys function to update MSYS2 package manager (pacman) before updating any other package to work around package manager's issues (e.g. msys2/MSYS2-packages#1962).

@MVoz
Copy link
Contributor

MVoz commented May 19, 2020

try checking the port to see if there will be an error
error: cannot remove /usr/bin/pacman.exe (Permission denied)

@emptyVoid
Copy link
Contributor Author

I tested it with vcpkg install icu[core], had no such error.

@JackBoosY
Copy link
Contributor

Waiting for merge #11459.

@MVoz
Copy link
Contributor

MVoz commented May 20, 2020

so it works locally for you, but in Ci ?

@emptyVoid
Copy link
Contributor Author

Could you please direct me to the said error message in CI? (afaict the logs of failed checks in this PR contain only some unrelated protobuf errors)

Copy link
Contributor

@JackBoosY JackBoosY left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please update the vesion info in ports/ffmpeg/CONTROL to test the changes?
See documentation.

@emptyVoid
Copy link
Contributor Author

Looks like MSYS2 team broke it once again -- msys2/MSYS2-packages#1962 (comment).

@JackBoosY
Copy link
Contributor

Waiting for upstream to provide a solution...

@emptyVoid
Copy link
Contributor Author

There are two possibilities:

  1. Manually download and update the package manager (I'm going to push it in a bit).
  2. Use MSYS2 from here https://github.com/msys2/msys2-installer/releases/ (not sure if these releases are persistent as I don't see any besides the latest).

@MVoz
Copy link
Contributor

MVoz commented May 21, 2020

wait for to removed symbolic links

#11483

@emptyVoid
Copy link
Contributor Author

Should probably go with #11471 as a member of MSYS2 confirms persistence of GitHub releases (msys2/MSYS2-packages#1962 (comment)).

@BillyONeal
Copy link
Member

error: could not open file /var/cache/pacman/pkg/msys2-runtime-3.1.4-2-x86_64.pkg.tar.zst: Child process exited with status 127
error: could not open file /var/cache/pacman/pkg/pacman-5.2.1-7-x86_64.pkg.tar.zst: Child process exited with status 127

Doesn't appear to have fixed the issue.

@BillyONeal
Copy link
Member

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@BillyONeal
Copy link
Member

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@BillyONeal
Copy link
Member

It looks like the current msys breakage causes our builders to become wedged afterwards so we need to accept this or #11471 after we've explicitly removed all the builders to get a clean pass.

@BillyONeal
Copy link
Member

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@BillyONeal BillyONeal requested a review from JackBoosY May 22, 2020 05:25
@emptyVoid
Copy link
Contributor Author

I browsed though failing logs after the force-push and it looked like as if CI continued from the state before force-push instead of continuing from the state before PR.

CI's new Windows runs have completed too quickly (apparently ffmpeg hasn't been rebuilt), should I bump its version once again?

@BillyONeal
Copy link
Member

@emptyVoid Given how everything related to msys is bricking builders right now I think we should choose whether we take this one or the other one before doing further investigation

@BillyONeal
Copy link
Member

Looks like this PR is working while the other one leaves ffmpeg broken:

PS D:\vcpkg2> .\vcpkg install ffmpeg
Your feedback is important to improve Vcpkg! Please take 3 minutes to complete our survey by running: vcpkg contact --survey
Computing installation plan...
A suitable version of cmake was not found (required v3.17.2). Downloading portable cmake v3.17.2...
Downloading cmake...
  https://github.com/Kitware/CMake/releases/download/v3.17.2/cmake-3.17.2-win32-x86.zip -> D:\vcpkg2\downloads\cmake-3.17.2-win32-x86.zip
Extracting cmake...
A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> D:\vcpkg2\downloads\7-zip.commandline.18.1.0.nupkg
Extracting 7zip...
A suitable version of nuget was not found (required v4.8.1). Downloading portable nuget v4.8.1...
Downloading nuget...
  https://dist.nuget.org/win-x86-commandline/v4.8.1/nuget.exe -> D:\vcpkg2\downloads\nuget.exe
The following packages will be built and installed:
    ffmpeg[avresample,core]:x86-windows
  * zlib[core]:x86-windows
Additional packages (*) will be modified to complete this operation.
A suitable version of powershell-core was not found (required v6.2.1). Downloading portable powershell-core v6.2.1...
Downloading powershell-core...
  https://github.com/PowerShell/PowerShell/releases/download/v6.2.1/PowerShell-6.2.1-win-x86.zip -> D:\vcpkg2\downloads\PowerShell-6.2.1-win-x86.zip
Extracting powershell-core...
Starting package 1/2: zlib:x86-windows
Building package zlib[core]:x86-windows...
-- Downloading http://www.zlib.net/zlib-1.2.11.tar.gz...
-- Extracting source D:/vcpkg2/downloads/zlib1211.tar.gz
-- Applying patch cmake_dont_build_more_than_needed.patch
-- Using source at D:/vcpkg2/buildtrees/zlib/src/1.2.11-deec42f53b
-- Configuring x86-windows
-- Building x86-windows-dbg
-- Building x86-windows-rel
-- Installing: D:/vcpkg2/packages/zlib_x86-windows/share/zlib/copyright
-- Performing post-build validation
-- Performing post-build validation done
Building package zlib[core]:x86-windows... done
Installing package zlib[core]:x86-windows...
Installing package zlib[core]:x86-windows... done
Elapsed time for package zlib:x86-windows: 4.579 s
Starting package 2/2: ffmpeg:x86-windows
Building package ffmpeg[avresample,core]:x86-windows...
-- Downloading https://github.com/ffmpeg/ffmpeg/archive/n4.2.tar.gz...
-- Extracting source D:/vcpkg2/downloads/ffmpeg-ffmpeg-n4.2.tar.gz
-- Applying patch 0001-create-lib-libraries.patch
-- Applying patch 0003-fix-windowsinclude.patch
-- Applying patch 0004-fix-debug-build.patch
-- Applying patch 0005-fix-libvpx-linking.patch
-- Applying patch 0006-fix-StaticFeatures.patch
-- Using source at D:/vcpkg2/buildtrees/ffmpeg/src/n4.2-a4d149a5c6
-- Downloading https://www.tortall.net/projects/yasm/snapshots/v1.3.0.6.g1962/yasm-1.3.0.6.g1962.exe...
-- Installing: D:/vcpkg2/downloads/tools/yasm/1.3.0.6/yasm.exe
-- Acquiring MSYS2...
-- Downloading https://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-base-x86_64-20190524.tar.xz/download...
gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
gpg: no ultimately trusted keys found
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
gpg: Generating pacman keyring master key...
gpg: key AFE006A04CB49425 marked as ultimately trusted
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/BC3BE574763C7F3450E1D34CAFE006A04CB49425.rev'
gpg: Done
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
==> Appending keys from msys2.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signing key D55E7A6D7CE9BA1587C0ACACF40D263ECA25678A...
  -> Locally signing key 123D4D51A1793859C2BE916BBBE514E53E0D0813...
  -> Locally signing key B91BCF3303284BF90CC043CA9F418C233E652008...
  -> Locally signing key 9DD0D4217D75A33B896159E6DA7EF2ABAEEA755C...
==> Importing owner trust values...
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: inserting ownertrust of 4
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   4  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   4  signed:   3  trust: 0-, 0q, 0n, 4m, 0f, 0u
gpg: depth: 2  valid:   3  signed:   0  trust: 3-, 0q, 0n, 0m, 0f, 0u
-- Downloading https://sourceforge.net/projects/msys2/files/REPOS/MSYS2/x86_64/libzstd-1.4.4-2-x86_64.pkg.tar.xz/download...
-- Downloading https://sourceforge.net/projects/msys2/files/REPOS/MSYS2/x86_64/zstd-1.4.4-2-x86_64.pkg.tar.xz/download...
-- Downloading https://sourceforge.net/projects/msys2/files/REPOS/MSYS2/x86_64/pacman-5.2.1-6-x86_64.pkg.tar.xz/download...
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) libzstd-1.4.4-2

Total Installed Size:  0.71 MiB

:: Proceed with installation? [Y/n]
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
installing libzstd...
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) zstd-1.4.4-2

Total Installed Size:  3.19 MiB

:: Proceed with installation? [Y/n]
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
installing zstd...
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) pacman-5.2.1-6

Total Installed Size:  37.74 MiB
Net Upgrade Size:      -8.86 MiB

:: Proceed with installation? [Y/n]
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
upgrading pacman...
:: Synchronizing package databases...
downloading mingw32.db...
downloading mingw32.db.sig...
downloading mingw64.db...
downloading mingw64.db.sig...
downloading msys.db...
downloading msys.db.sig...
:: Starting core system upgrade...
warning: resolving dependencies...
terminate other MSYS2 programs before proceeding
looking for conflicting packages...

Packages (6) bash-4.4.023-2  filesystem-2020.02-2  mintty-1~3.1.4-1  msys2-runtime-3.1.4-2  pacman-5.2.1-7  pacman-mirrors-20200329-1

Total Download Size:   19.40 MiB
Total Installed Size:  62.51 MiB
Net Upgrade Size:       1.65 MiB

:: Proceed with installation? [Y/n]
:: Retrieving packages...
downloading msys2-runtime-3.1.4-2-x86_64.pkg.tar.zst...
downloading bash-4.4.023-2-x86_64.pkg.tar.xz...
downloading filesystem-2020.02-2-x86_64.pkg.tar.xz...
downloading mintty-1~3.1.4-1-x86_64.pkg.tar.xz...
downloading pacman-mirrors-20200329-1-any.pkg.tar.xz...
downloading pacman-5.2.1-7-x86_64.pkg.tar.zst...
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
upgrading msys2-runtime...
upgrading bash...
upgrading filesystem...
upgrading mintty...
upgrading pacman-mirrors...
upgrading pacman...
warning: terminate MSYS2 without returning to shell and check for updates again
warning: for example close your terminal window instead of calling exit
-- Acquiring MSYS2... OK
-- Acquiring MSYS Packages...
-- Acquiring MSYS Packages... OK
-- Building Options: --enable-asm --enable-yasm --disable-doc --enable-debug --enable-runtime-cpudetect --disable-openssl --disable-ffmpeg --disable-ffplay --disable-ffprobe --disable-libvpx --disable-libx264 --disable-opencl --disable-lzma --disable-bzlib --enable-avresample --disable-cuda --disable-nvenc --disable-cuvid --disable-libnpp  --disable-static --enable-shared --extra-cflags=-DHAVE_UNISTD_H=0
-- Building Release Options:  --extra-cflags=-MD --extra-cxxflags=-MD
-- Building ffmpeg for Release

@JackBoosY JackBoosY added the info:needs-maintainer-attention Lets the current 'on rotation' vcpkg maintainer know they need to look at this. label May 22, 2020
Copy link
Member

@vicroms vicroms left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BillyONeal
Copy link
Member

Given that this fixes CI I'm going to merge this one, that does not mean we aren't necessarily interested in the more complete updates in the other one.

@BillyONeal
Copy link
Member

Thanks for your contribution!

@BillyONeal BillyONeal merged commit e792d1c into microsoft:master May 22, 2020
@emptyVoid emptyVoid deleted the fix-issue-11438 branch May 25, 2020 05:26
@JackBoosY
Copy link
Contributor

JackBoosY commented Jun 5, 2020

Hi @emptyVoid, it seems broken again:

error: could not open file /var/cache/pacman/pkg/binutils-2.34-3-x86_64.pkg.tar.zst: Child process exited with status 127

Could you please check that?

Thanks,

@emptyVoid
Copy link
Contributor Author

Hi @JackBoosY, could you elaborate, please, on how to reproduce the issue?

I've just tested vcpkg install icu[core] on a clean install with current master -- no errors.
And vcpkg install x264 installs binutils-2.34-3-x86_64.pkg.tar.zst without issue (even if the port itself fails to configure later on).

If I had to guess I'd say the issue manifests on some non-clean install where MSYS2 was installed before the commotion with package compression scheme change.

I think this could be fixed by updating MSYS2 installation in vcpkg_acquire_msys before trying to install new packages. Although I'm not sure what ramifications the update could impose on packages already built with MSYS2.

And it looks like the workaround this PR added is no longer needed:
msys2/MSYS2-packages#1962 (comment)

@JackBoosY
Copy link
Contributor

@emptyVoid I can repro this issue by installing x264:x86-windows:

warning: diffutils-3.7-1 is up to date -- skipping
warning: pkg-config-0.29.2-1 is up to date -- skipping
warning: libtool-2.4.6-9 is up to date -- skipping
warning: gettext-0.19.8.1-1 is up to date -- skipping
warning: gettext-devel-0.19.8.1-1 is up to date -- skipping
warning: make-4.3-1 is up to date -- skipping
error: could not open file /var/cache/pacman/pkg/binutils-2.34-3-x86_64.pkg.tar.zst: Child process exited with status 127
error: failed to commit transaction (cannot open package file)

If these changes are not needed, please open a new PR to remove them.

Thanks.

@JackBoosY
Copy link
Contributor

After remove downloads/tools/msys2 and rebuild x264:x86-windows:

Unknown option --build=i686-pc-mingw32, ignored
Unknown option --target=i686-pc-mingw32, ignored
Unknown option --disable-silent-rules, ignored
Unknown option --verbose, ignored
Unknown option --disable-static, ignored
Unknown option --sbindir=${prefix}/../tools/x264/debug/sbin, ignored
Unknown option --datarootdir=${prefix}/share/x264, ignored
No working C compiler found.

If the old msys files may cause errors after upgrade, we should install msys in a directory named after the version number.

@emptyVoid
Copy link
Contributor Author

The version number of MSYS2 won't really help since the trouble comes from new packages and not from MSYS2 itself.

In my opinion the troublesome part of vcpkg_acquire_msys is the following string:

COMMAND ${PATH_TO_ROOT}/usr/bin/bash.exe --noprofile --norc -c "pacman -Sy --noconfirm --needed ${_am_PACKAGES}"

the -Sy key instructs pacman to update its package database before installing requested packages. Because of it an old pacman then tries to install a package with an unsupported compression.

I see a few ways how to deal with it:

  1. Remove -y option leaving just -S -- this would instruct pacman to operate on the package database it got when MSYS2 was initially installed.
  2. Update pacman before trying to install new MSYS2 packages.
  3. Update pacman before trying to install new MSYS2 packages, and rebuild the MSYS2 dependent ports which are already installed.
  4. Use separate MSYS2 for each dependent port.

Approaches (1) and (3) guarantee all the MSYS2 dependent ports would get built with the same version of MSYS2 packages.
Whereas (2) and (4) might cause the situation when some port (A), dependent on MSYS2 package (m) and on some other port (B) which in turn also dependent on package (m), would be built with a newer version of (m) than the one (B) was built with.
(3) would probably require adjustments to vcpkg, as I believe it doesn't track port dependencies to build tools.
(4) would result in the bloating of build trees -- at least ~400MiB per MSYS2 installation (with package archives cached in a shared location).

@JackBoosY JackBoosY added the category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly label Jun 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly info:needs-maintainer-attention Lets the current 'on rotation' vcpkg maintainer know they need to look at this.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[icu[core]:x64-windows] build failure

5 participants