Add intial runfile installer code written by David Bielecki.

[pbhandar-amd]

Motivation

Add code for the runfile installer which is going to be another install method, alongside tarball, python wheels and package manager.
The installer provides a single ROCm installer for all-gfx archs, handling dependency install as well as post-installation for setting up ROCm, and other installation options from a single unified .run file. Provided ROCm/amdgpu dependencies are installed on a system, the runfile installer may be used for offline install of ROCm and/or the amdgpu driver.
The runfile installer is an installation method available for installing ROCm and/or the amdgpu driver in the legacy ROCm releases (https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/rocm-runfile-installer.html). We are adding support for ROCm releases from TheRock starting with re-introduction of the installer in 7.12 (ROCM-1861).

Technical Details

File changes/additions:

• build_tools/packaging/runfile-installer (addition to https://github.com/ROCm/TheRock/tree/main/build_tools/packaging)
  • runfile-installer - the installer project, which includes the main build script : build-runfile-installer.sh for building the install .run.
  • runfile-installer/build-installer
    • Contains the build scripts used by build-runfile-installer.sh for setting up and building the output .run file for the installer.
  • runfile-installer/package-extractor
    • Contains the package-extractor scripts used in the build pipeline to extract packages used for the content of the installer ie. core-sdk
      • we use the current packages built by TheRock as a source for the installer (this implementation carried forward from the legacy installer)
  • runfile-installer/package-puller
    • Pulls/downloads the packages needed for installer (core-sdk / gfxXYZ).
  • runfile-installer/rocm-installer
    • Main installer which will contain all the contents needed for installation of ROCm and amdgpu driver, as well as the GUI frontend.
  • runfile-installer/UI
    • Source for the GUI frontend interface to the runfile installer.
    • GUI is optional and users can use the command-line interface to the installer if they choose.

The Build:
- Run the build-runfile-installer script with args required for the type of installer build job: dev, nightly, prerelease
- Builds are run using the ManyLinux docker from TheRock, locally or via build jobs being developed for TheRock.

build-runfile-installer.sh
	|
	—> setup-installer —> package-puller —> build-installer —> package-extractor —> GUI build —> .run create (makeself)

Test Plan

• For legacy runfile installer, built runfile installers were tested by SQA and the Installer team. The plan is the same for TheRock version.
• The installer testing is performed across the 15+ Linux distros supported by ROCm using internal validation scripts.

• For TheRock, we would like to add a build test to validate the construction of the runfile installer .run file.

Test Result

Build pass:
- Is the file rocm-installer.run created

Build failure:
- Package-pull failure - failed to download packages need to build the installer.
- Package-extraction failure - failed to install tools need to extract packages or failure during extraction.
- GUI build failure
- makeself failure when constructing the rocm-installer .run file

Install pass:
- ROCm and amdgpu installs without error using command-line or GUI interfaces to the installer
- Basic validation passes of ROCm/amdgpu

Install failure:
- ROCm or amdgpu fails during install from command-line or GUI
- ROCm or amdgpu installs, but is not functional.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[ScottTodd]

Please fill in the PR description. I'm not going to look at this until it meets that bare minimum bar.

[stellaraccident]

No tests? How are we supposed to maintain this?

Separately: it looks like you leaked gpg keys. Please explain to me internally how this is ok and whether we need to rotate anything.

[astrelsky]

No tests? How are we supposed to maintain this?

Separately: it looks like you leaked gpg keys. Please explain to me internally how this is ok and whether we need to rotate anything.

$ gpg --list-packets rocm-keyring.gpg
# off=0 ctb=99 tag=6 hlen=3 plen=525
:public key packet:
        version 4, algo 1, created 1470083360, expires 0
        pkey[0]: [4096 bits]
        pkey[1]: [17 bits]
        keyid: 9386B48A1A693C5C
# off=528 ctb=b4 tag=13 hlen=2 plen=40
:user ID packet: "AMD MLSE DevOps <dl.MLSE.DevOps@amd.com>"
# off=570 ctb=89 tag=2 hlen=3 plen=574
:signature packet: algo 1, keyid 9386B48A1A693C5C
        version 4, created 1643876755, md5len 0, sigclass 0x13
        digest algo 2, begin of digest 8d 40
        hashed subpkt 27 len 1 (key flags: 03)
        hashed subpkt 11 len 5 (pref-sym-algos: 9 8 7 3 2)
        hashed subpkt 21 len 5 (pref-hash-algos: 8 2 9 10 11)
        hashed subpkt 22 len 3 (pref-zip-algos: 2 3 1)
        hashed subpkt 30 len 1 (features: 01)
        hashed subpkt 23 len 1 (keyserver preferences: 80)
        hashed subpkt 2 len 4 (sig created 2022-02-03)
        hashed subpkt 9 len 4 (key expires after 10y186d11h56m)
        subpkt 16 len 8 (issuer key ID 9386B48A1A693C5C)
        data: [4096 bits]
# off=1147 ctb=b9 tag=14 hlen=3 plen=525
:public sub key packet:
        version 4, algo 1, created 1470083360, expires 0
        pkey[0]: [4096 bits]
        pkey[1]: [17 bits]
        keyid: 30C07AF01A6D36BA
# off=1675 ctb=89 tag=2 hlen=3 plen=549
:signature packet: algo 1, keyid 9386B48A1A693C5C
        version 4, created 1643876791, md5len 0, sigclass 0x18
        digest algo 10, begin of digest c0 1c
        hashed subpkt 27 len 1 (key flags: 0C)
        hashed subpkt 2 len 4 (sig created 2022-02-03)
        hashed subpkt 9 len 4 (key expires after 10y186d11h57m)
        subpkt 16 len 8 (issuer key ID 9386B48A1A693C5C)
        data: [4096 bits]

$ gpg --list-packets rocm-internal.gpg
# off=0 ctb=99 tag=6 hlen=3 plen=269
:public key packet:
        version 4, algo 1, created 1664219662, expires 0
        pkey[0]: [2048 bits]
        pkey[1]: [17 bits]
        keyid: AC344D255E48C714
# off=272 ctb=b4 tag=13 hlen=2 plen=59
:user ID packet: "compute-artifactory (Signing key) <jenkins-compute@amd.com>"
# off=333 ctb=89 tag=2 hlen=3 plen=313
:signature packet: algo 1, keyid AC344D255E48C714
        version 4, created 1664219662, md5len 0, sigclass 0x13
        digest algo 2, begin of digest 50 32
        hashed subpkt 2 len 4 (sig created 2022-09-26)
        hashed subpkt 27 len 1 (key flags: 03)
        hashed subpkt 11 len 6 (pref-sym-algos: 9 8 7 3 2 1)
        hashed subpkt 21 len 5 (pref-hash-algos: 8 2 9 10 11)
        hashed subpkt 22 len 3 (pref-zip-algos: 2 3 1)
        hashed subpkt 30 len 1 (features: 01)
        hashed subpkt 23 len 1 (keyserver preferences: 80)
        subpkt 16 len 8 (issuer key ID AC344D255E48C714)
        data: [2047 bits]
# off=649 ctb=b9 tag=14 hlen=3 plen=269
:public sub key packet:
        version 4, algo 1, created 1664219662, expires 0
        pkey[0]: [2048 bits]
        pkey[1]: [17 bits]
        keyid: 8165C203DA3B8702
# off=921 ctb=89 tag=2 hlen=3 plen=287
:signature packet: algo 1, keyid AC344D255E48C714
        version 4, created 1664219662, md5len 0, sigclass 0x18
        digest algo 2, begin of digest b1 98
        hashed subpkt 2 len 4 (sig created 2022-09-26)
        hashed subpkt 27 len 1 (key flags: 0C)
        subpkt 16 len 8 (issuer key ID AC344D255E48C714)
        data: [2048 bits]

Hopefully this helps answer that question @stellaraccident

[stellaraccident]

$ gpg --list-packets rocm-keyring.gpg
# off=0 ctb=99 tag=6 hlen=3 plen=525
:public key packet:
        version 4, algo 1, created 1470083360, expires 0
        pkey[0]: [4096 bits]
        pkey[1]: [17 bits]
        keyid: 9386B48A1A693C5C
# off=528 ctb=b4 tag=13 hlen=2 plen=40
:user ID packet: "AMD MLSE DevOps <dl.MLSE.DevOps@amd.com>"
# off=570 ctb=89 tag=2 hlen=3 plen=574
:signature packet: algo 1, keyid 9386B48A1A693C5C
        version 4, created 1643876755, md5len 0, sigclass 0x13
        digest algo 2, begin of digest 8d 40
        hashed subpkt 27 len 1 (key flags: 03)
        hashed subpkt 11 len 5 (pref-sym-algos: 9 8 7 3 2)
        hashed subpkt 21 len 5 (pref-hash-algos: 8 2 9 10 11)
        hashed subpkt 22 len 3 (pref-zip-algos: 2 3 1)
        hashed subpkt 30 len 1 (features: 01)
        hashed subpkt 23 len 1 (keyserver preferences: 80)
        hashed subpkt 2 len 4 (sig created 2022-02-03)
        hashed subpkt 9 len 4 (key expires after 10y186d11h56m)
        subpkt 16 len 8 (issuer key ID 9386B48A1A693C5C)
        data: [4096 bits]
# off=1147 ctb=b9 tag=14 hlen=3 plen=525
:public sub key packet:
        version 4, algo 1, created 1470083360, expires 0
        pkey[0]: [4096 bits]
        pkey[1]: [17 bits]
        keyid: 30C07AF01A6D36BA
# off=1675 ctb=89 tag=2 hlen=3 plen=549
:signature packet: algo 1, keyid 9386B48A1A693C5C
        version 4, created 1643876791, md5len 0, sigclass 0x18
        digest algo 10, begin of digest c0 1c
        hashed subpkt 27 len 1 (key flags: 0C)
        hashed subpkt 2 len 4 (sig created 2022-02-03)
        hashed subpkt 9 len 4 (key expires after 10y186d11h57m)
        subpkt 16 len 8 (issuer key ID 9386B48A1A693C5C)
        data: [4096 bits]

$ gpg --list-packets rocm-internal.gpg
# off=0 ctb=99 tag=6 hlen=3 plen=269
:public key packet:
        version 4, algo 1, created 1664219662, expires 0
        pkey[0]: [2048 bits]
        pkey[1]: [17 bits]
        keyid: AC344D255E48C714
# off=272 ctb=b4 tag=13 hlen=2 plen=59
:user ID packet: "compute-artifactory (Signing key) <jenkins-compute@amd.com>"
# off=333 ctb=89 tag=2 hlen=3 plen=313
:signature packet: algo 1, keyid AC344D255E48C714
        version 4, created 1664219662, md5len 0, sigclass 0x13
        digest algo 2, begin of digest 50 32
        hashed subpkt 2 len 4 (sig created 2022-09-26)
        hashed subpkt 27 len 1 (key flags: 03)
        hashed subpkt 11 len 6 (pref-sym-algos: 9 8 7 3 2 1)
        hashed subpkt 21 len 5 (pref-hash-algos: 8 2 9 10 11)
        hashed subpkt 22 len 3 (pref-zip-algos: 2 3 1)
        hashed subpkt 30 len 1 (features: 01)
        hashed subpkt 23 len 1 (keyserver preferences: 80)
        subpkt 16 len 8 (issuer key ID AC344D255E48C714)
        data: [2047 bits]
# off=649 ctb=b9 tag=14 hlen=3 plen=269
:public sub key packet:
        version 4, algo 1, created 1664219662, expires 0
        pkey[0]: [2048 bits]
        pkey[1]: [17 bits]
        keyid: 8165C203DA3B8702
# off=921 ctb=89 tag=2 hlen=3 plen=287
:signature packet: algo 1, keyid AC344D255E48C714
        version 4, created 1664219662, md5len 0, sigclass 0x18
        digest algo 2, begin of digest b1 98
        hashed subpkt 2 len 4 (sig created 2022-09-26)
        hashed subpkt 27 len 1 (key flags: 0C)
        subpkt 16 len 8 (issuer key ID AC344D255E48C714)
        data: [2048 bits]

Thanks. So public keys only. I would have preferred that this was described in the PR description.

[astrelsky]

Thanks. So public keys only. I would have preferred that this was described in the PR description.

Better safe than sorry, especially when they have internal in their name.

[dbieleck]

The GPG keys will be removed. It terms of the testing, we have some test scripts that we use as part of the BM testing, which are run on the 15+ Distros we are documenting as supported by ROCm. As part of this testing, we validate if the install was successful, and base functionality is working. @stellaraccident are you looking for tests for just validating the build? We're not sure what's done in terms of the rpm/deb package build/generation and what's tested there. For the runfile installer, we could do something similar to the package testing. In terms of maintenance, myself and my team would be doing that ie. add new features to the installer, bug fixing, adding new distros, updating the GUI frontend, etc. This goes back to something I was wondering, if it makes sense to just have the 1 or 2 base build scripts in TheRock and have the installer parts and GUI source in a separate repo that are cloned as part of the build. We're open to suggestions...

[ScottTodd]

I'm adding the "skip-ci" label to this PR for now, since these new files are not referenced by any CI workflows and continuous syncing of this branch keeps retriggering (expensive) CI workflows.

I see that the PR has a description now, but this sort of large code dump is going to take significant time to review from design, policy, and code perspectives. I ran some automated code review tooling on this PR offline and it quickly spotted many style guide violations (https://github.com/ROCm/TheRock/tree/main/docs/development/style_guides), testing gaps (no unit tests), and security issues (command injection, out of bounds access, buffer overflows, missing error handling, etc.).

I don't think we'll be able to accept this code in a bulk drop in its current form (especially given the confusing authorship in the PR title - code needs a clear owner, it isn't just written once by one person, added to a repo by another, and then called finished), but finding the right contribution and maintenance model is beyond the scope of PR review threads - may need a higher bandwidth office hours discussion.