diff --git a/Makefile b/Makefile index ad3f0958..f92dd172 100644 --- a/Makefile +++ b/Makefile @@ -24,12 +24,7 @@ default: @echo 'This Makefile is only used for cleaning the repository and' @echo 'running the style checks. To build packages, first run' - @echo './setup.sh, then run ./buildlist.sh by passing the' - @echo 'appropriate package list. For instance to build the "userland"' - @echo 'package list, run:' - @echo ' ./setup.sh && ./buildlist.sh userland' - @echo 'You can also build a single package with:' - @echo ' ./setup.sh && ./buildpkg.sh ' + @echo './setup.sh, then run ./buildpkg.sh . @echo 'Refer to the README for more info.' clean: @@ -37,7 +32,6 @@ clean: @rm -rf artifacts @rm -f *.buildinfo *.changes *.deb @rm -rf update-status - @build-info-pkg/clean.sh shellcheck: shellcheck --exclude=SC1090,SC1091 \ diff --git a/README.md b/README.md index 3bd33d7b..ec98f80a 100644 --- a/README.md +++ b/README.md @@ -14,13 +14,14 @@ projects. 1. [Environment Variables](#environment-variables) 1. [Package Definition](#package-definition) * [Package Variables](#package-variables) - * [Package Hooks](#package-hooks) + * [Package Hooks](#package-stages-and-hooks) * [Package Environment Variables](#package-environment-variables) * [Package WORKDIR](#package-workdir) 1. [Adding New Packages](#adding-new-packages) * [Third-party package](#third-party-package) * [In-house package](#in-house-package) 1. [Testing your changes](#testing-your-changes) +1. [Package Lists](#package-lists) 1. [Versions and Branches](#versions-and-branches) 1. [Contributing](#contributing) 1. [Statement of Support](#statement-of-support) @@ -64,7 +65,7 @@ cd linux-pkg ./setup.sh ``` -### Step 3. Build a package or a package list +### Step 3. Build a package We can now build an arbitrary package. Any package in the [packages directory](./packages) would do. Let's pick `cloud-init` as an @@ -74,34 +75,22 @@ example: ./buildpkg.sh cloud-init ``` -Packages will be stored in directory `packages/cloud-init/tmp/artifacts/`. - -Linux-pkg also allows you to build a pre-defined list of packages. Package -lists can be found in the [package-lists directory](./package-lists/build/). -Let's try to build the "userland" package list: - -``` -./buildlist.sh userland -``` - -Note that the build of the "userland" list can take close to an hour. The -artifacts for all the packages built will be located in directory `artifacts/`. +Build artifacts will be stored in directory +`packages/cloud-init/tmp/artifacts/`. ## Project Summary -There are two main tasks that are performed by this framework: building lists -of packages so that they can be later included in -[appliance-build](https://github.com/delphix/appliance-build), and keeping each -package up-to-date with its upstream project by updating the appropriate git -branches. +There are two main tasks that are performed by this framework: building +packages and keeping each package up-to-date with its upstream project by +updating the appropriate git branches. ### Building packages -This task is relatively straight forward. Every package listed in the target -package list is built and a metapackage is created with the build info of each -package that was built. -You can see section [Scripts > buildlist.sh](#buildlistsh) below for more -details. +This task is relatively straight forward. What linux-pkg calls a "package" is +really a project (usually a git project) that has a build recipe and that +produces one or more debian packages and some other metadata files. + +See [Scripts > updatelist.sh](#buildpkgsh) below. ### Updating third-party packages @@ -128,34 +117,42 @@ of the package tuned to work with our Ubuntu distribution. When updating a package, we first check if the **upstreams/master** branch is up-to-date, by fetching the latest version of the upstream git repository or the -Ubuntu source package. If changes are detected, we update **upstreams/master**. +Ubuntu source package. If changes are detected, we update the +**upstreams/master** branch and push the changes to GitHub. The second step is to check if the **master** branch is up-to-date with **upstreams/master**. If it is already up-to-date, then we are done. If not, then we attempt merging **upstreams/master** into **master**. -If the merge is successful, then we attempt building the **master** branch. The -merge is considered failed if the build fails, which means that the **master** -branch of the main repository will not be updated. +If the merge is successful, then we push the changes to a staging branch on +GitHub, called **projects/auto-update/master/merging**. The intent is for +a different system to fetch those changes, build them, and then launch tests. -Note that any updates are pushed independently to the **upstreams/master** and -**master** branches of the Delphix repository for the package. +See [Scripts > sync-with-upstream.sh](#sync-with-upstreamsh) below. -Although for now we only support auto-updating the **master** branch, the -framework is designed so that other branches could also be auto-updated. +Once the merge has been tested, [Scripts > push-merge.sh](#push-mergesh) is +called on the original VM to push the changes to the **master** branch on +GitHub. -For additional details, you can see section -[Scripts > updatelist.sh](#updatelistsh) below. +Note that the example above targets the **master** branch, but the same +workflow could apply to other branches, like **6.0/stage**, although it is +not currently in use. ## Scripts A set of scripts were created in this repository to allow easily building and updating packages both manually and through automation (e.g. Jenkins). +### query-packages.sh + +This script can be called on most unix-based systems to query metadata on the +packages built by linux-pkg. This script does not install anything on the +system, so it can be run anywhere without any side effects. + ### setup.sh Installs dependencies for the build framework. Needs to be run once to configure -the system, before any other scripts. +the system, before any other scripts (except query-packages.sh). ### buildpkg.sh @@ -167,97 +164,71 @@ Builds a single package. Package name must match a directory under ``` The build will look at `packages//config.sh` for instructions on where -to fetch the package from and how build it. The build will be performed in +to fetch the package from and how to build it. The build will be performed in `packages//tmp/`, and build artifacts for this package will be stored in the `artifacts` sub-directory. -`buildpkg.sh` includes additional options. The most common of them is `-u`, -which will update the package with upstream. See section -[Updating third-party packages](#updating-third-party-packages) for more info. - -### buildlist.sh +Note that if the build of the package depends on build artifacts from another +linux-pkg package, those will be fetched from a predetermined S3 location. -Builds a list of packages and the assossiated metapackage. The list of packages -is read from [package-lists/build/{list}.pkgs](./package-lists/build/) and -builds the packages listed there by invoking `buildpkg.sh` on each one of them. -Once they are all built, it builds the metapackage, which stores the build info -for each package that was built -- info such as git hash, git url and git -branch. +### checkupdates.sh +Usage: ``` -./buildlist.sh +./checkupdates.sh ``` -`buildlist.sh` doesn't take build options but can be configured by passing -various environment variables. -See section [Environment Variables](#environment-variables) for more details. +This checks if a package has updates in the upstream project that haven't been +pulled into the **upstreams/master** branch, or if the **upstreams/master** +branch has commits that haven't been merged into the **master** branch. -Packages are built sequentially in the order defined in the pkgs file. If a -package fails to build, the script exits immediately. +If updates are available, the file `/update-available` will be created. -### jenkins-buildlist.sh +The intention of this script is to inform the caller whether an update job +should be called for the given package. -This is a wrapper script around `buildlist.sh` and was designed -to be called by Jenkins. Any environment variables that are passed to -`jenkins-buildlist.sh` are propagated to the child script. In addition, -`jenkins-buildlist.sh` interprets environment variables specified in section -[Environment variables specific to jenkins-buildlist](#environment-variables-specific-to-jenkins-buildlist). - -### updatelist.sh - -Updates all the packages listed in package list -[package-lists/update/{list}.pkgs](./package-lists/update/): +### sync-with-upstream.sh +Usage: ``` -./updatelist.sh [-n] +./sync-with-upstream.sh ``` -Here are the steps for updating one package: - -1. Run `buildpkg.sh -u `. This will attempt to update the - **upstreams/master** branch, and then attempt to merge **upstreams/master** - into **master**. If changes are detected on **master**, then the package will - be built. If a package is listed in - [package-lists/auto-merge-blacklist.pkgs](./package-lists/auto-merge-blacklist.pkgs), - then `-M` will be passed to `buildpkg.sh` and we will not attempt updating - **master**. - -1. If changes are detected for **upstreams/master** or **master**, push them to - the default repository for the package (e.g. `github.com/delphix/`). - This is done by invoking `push-updates.sh` for each branch. Note that - **upstreams/master** will be updated even if merge with **master** failed, - allowing developers to later perform the merge manually. +This script has 2 tasks: +1. Check if the upstream project has updates that are not pulled into the +**upstreams/master** branch of the package, and if so then update that branch +and push changes to GitHub. +2. Merge **upstreams/master** into **master** and push the changes to a staging +branch on GitHub, called **projects/auto-update/master/merging**. Another +system should use that branch to build the package, and then run the appropriate +integration tests. -Each package is processed independently, so a failure to update one package -doesn't affect the update of other packages. A report is generated at the end. +After testing has been completed, `push-merge.sh ` should be called on +the same system to push the merge to the **master** branch. -`updatelist.sh` can be run in dry-run mode by passing `-n` so that package -updates are not pushing to the target repository. Its behaviour can also be -configured by passing various environment variables. See section -[Environment Variables](#environment-variables) for more details. +Note that the DRYRUN environment variable must be set when running this script. +If DRYRUN is set to "true", then changes are not pushed to GitHub in step 1, +and staged changes are pushed to **projects/auto-update/master/merging-dryrun** +in step 2 instead of the non-dryrun branch. The intention is that when testing +changes to the logic we want to be able to run most of the logic, but without +affecting the production branches. -### jenkins-updatelist.sh - -This is a wrapper script around `updatelist.sh` and was designed to be called -by Jenkins. Any environment variables that are passed to `jenkins-buildlist.sh` -are propagated to the child script. - -### push-updates.sh - -This script pushes branch updates to the default repository for the package. It -should be called after running `buildpkg.sh -u `. The script should be -invoked with: +### push-merge.sh +Usage: ``` -./push-update.sh -u|-m +./push-merge.sh ``` -Running it with `-u` will update **upstreams/master** and running it with `-m` -will update **master**. +This must be called on a system that has previously called +`sync-with-upstream.sh` for the same package. It will push the merge that was +previously prepared by `sync-with-upstream.sh` to the production **master** +branch, after checking that the **master** branch hasn't been modified since +`sync-with-upstream.sh` was called. -Note that credentials for a user that has permissions to push to the target -repository must be passed. Passing the `-n` option does a dry-run, meaning that -the target repository won't be updated (`-n` will be passed to `git push`). +Like for `sync-with-upstream.sh`, the DRYRUN environment variable must be set +to run this script. However, the script will fail unless DRYRUN is set to +"false" given that there is not much that can be tested in dry-run mode. ## Environment Variables @@ -268,61 +239,75 @@ of some of the scripts defined above. we are running on the appropriate Ubuntu distribution in AWS. Affects all scripts. -* **DRY_RUN**: Set to "true" to prevent `updatelist.sh` from updating production - package repositories. `updatelist.sh` will invoke `push-updates.sh` with `-n`. +* **DRYRUN**: Must be set to either "true" of "false" when running script + [sync-with-upstream.sh](#sync-with-upstreamsh), and to "false" when running + script [push-merge.sh](#push-mergesh). * **PUSH_GIT_USER, PUSH_GIT_PASSWORD**: Set to the git credentials used to push - updates to package repositories. Affects `updatelist.sh` and - `push-updates.sh`. + updates to package repositories. Affects scripts + [sync-with-upstream.sh](#sync-with-upstreamsh) and + [push-merge.sh](#push-mergesh). * **DEFAULT_REVISION**: Default revision to use for packages that do not have a revision defined. If not set, it will be auto-generated from the timestamp. - Applies to `buildpkg.sh` and `buildlist.sh`. - -* **DEFAULT_BRANCH**: Default git branch to use when fetching a package that - does not have a branch explicitly defined. If not set, it will default to - "master". Applies to `buildpkg.sh` and `buildlist.sh`. - -* **CHECKSTYLE**: Applies to `buildlist.sh`. Passes `-c` to `buildpkg.sh` when - `CHECKSTYLE` is "true" to execute the `checkstyle` hook when building a package. - See [Package Definition](#package-definition) section for more details about - the hook. - -* **TARGET_PLATFORMS**: Some packages build kernel modules. This specifies which - kernel versions to build those packages for and accepts a space-separated list - of values. If the value is a platform, such as "aws" or "generic", then it - will auto-determine the default kernel version for the provided platform. If - `TARGET_PLATFORMS` is unset or "default", then it will build for all supported - platforms. - -* **UPDATE_PACKAGE_NAME**: Applies to `updatelist.sh` only. If this variable is - set then `updatelist.sh` only updates the package specified by this variable. - -* **{PACKAGE}_GIT_URL, {PACKAGE}_GIT_BRANCH, {PACKAGE}_VERSION, - {PACKAGE}_REVISION**: Can be used to override defaults for a given package. - `{PACKAGE}` is the package name in upper case with `-` converted to `_`. For + Applies to [buildpkg.sh](#buildpkgsh). + +* **DEFAULT_GIT_BRANCH**: The product branch that is being built or updated is + typically stored in the file `branch.config`, however it can be overridden via + DEFAULT_GIT_BRANCH. The product branch is used in multiple instances. When + running [setup.sh](#setupsh), it will determine what linux-package-mirror + link to use when fetching packages from apt (although those links can be + overridden via DELPHIX_PACKAGE_MIRROR_MAIN and + DELPHIX_PACKAGE_MIRROR_SECONDARY). When running [buildpkg.sh](#buildpkgsh), + it will determine which branch to fetch from the package's repository, unless + it is overridden via `-b`; if the package has build-dependencies on other + linux-pkg packages, those dependencies will be fetched from an S3 url that is + versioned based on the product branch (although the package dependencies + URLs can be overridden via package_S3_URL variables). Finally, when running + [sync-with-upstream.sh](#sync-with-upstreamsh) or + [push-merge.sh](#push-mergesh) it defines what branch of the package is + being updated. + +* **TARGET_KERNEL_FLAVOURS**: Some packages have build dependencies on the + linux kernel. Those packages have `PACKAGE_DEPENDENCIES="@linux-kernel"` in + their `config.sh`. By default, those packages are built for all the supported + kernel flavours (see SUPPORTED_KERNEL_FLAVORS in `common.sh`), however it is + possible to restrict which kernel flavours those packages are built for. + +* **package_GIT_URL, package_GIT_BRANCH, package_VERSION, + package_REVISION**: Can be used to override defaults for a given package. + `package` is the package name in upper case with `-` converted to `_`. For instance `CLOUD_INIT_GIT_BRANCH=feature1` would set the branch to fetch package `cloud-init` from to `feature1`. This is useful when running `buildlist.sh` to override defaults for multiple packages. Applies to both `buildlist.sh` and `buildpkg.sh`. -### Environment variables specific to jenkins-buildlist - -* **SINGLE_PACKAGE_NAME**: When running `jenkins-buildlist.sh`, other - `SINGLE_PACKAGE_{*}` parameters mentioned below are used to override the - defaults for this package. - -* **SINGLE_PACKAGE_GIT_URL, SINGLE_PACKAGE_GIT_BRANCH, SINGLE_PACKAGE_VERSION, - SINGLE_PACKAGE_REVISION**: Applies to `jenkins-buildlist.sh` only. Those are - equivalent to the `{PACKAGE}_{*}` variables described previously but apply to - the package passed in `SINGLE_PACKAGE_NAME`. They are added for convenience - when using Jenkins. - -* **CUSTOM_BUILDER_ENV**: Applies to `jenkins-buildlist.sh` only. This is a - multi-line field that takes one `{PACKAGE}_{*}=value` entry per line and is - parsed by `jenkins-buildlist.sh` to set the specified `{PACKAGE}_{*}` - environment variables. This can be used to set any number of `{PACKAGE}_{*}` - variables from Jenkins. +* **package_S3_URL**: Similar to the package_VAR variables above. This is used + to override the default S3 location for where package build-dependencies are + fetched for a given linux-pkg package. For instance, if you are building + bpftrace, which has `PACKAGE_DEPENDENCIES="bcc"` in its config, the + `fetch_dependencies()` stage in the build will fetch the latest build + artifacts of the bcc package from a predetermined S3 location. If you pass + `BCC_S3_URL=s3://path/to/custom/bcc/artifacts` then those artifacts will be + fetched insteasd. + +* **DELPHIX_PACKAGE_MIRROR_MAIN, DELPHIX_PACKAGE_MIRROR_SECONDARY**: When + the [setup.sh](#setupsh) script is run, it will configure the apt sources + to point to versioned delphix mirrors of the Ubuntu archive (MAIN mirror) + and of some auxiliary archives (SECONDARY mirror). Delphix has many snapshots + of those mirrors at different points in time, and if you want to use a custom + snapshot, you can pass it in those environment variables. + +* **JENKINS_OPS_DIR**: When fetching artifacts from other linux-pkg packages + that are marked as dependencies of a package, by default we look for a + specific S3 path that contains production package artifacts generated by + post-push jobs of the ops Jenkins agent. The production ops Jenkins agent + stores artifacts in the special `jenkins-ops` sub-directory. When using + a developer ops Jenkins agent, it stores build artifacts in a different S3 + sub-directory: `jenkins-ops.`. By setting JENKINS_OPS_DIR to that + sub-directory you can instruct linux-pkg to fetch artifacts of build + dependencies produced by the developer Jenkins instance instead of the + production one. ## Package Definition @@ -344,6 +329,13 @@ Here is a list of variables that can be defined for a package: `https://` URL. One exception is if the source of the package being built isn't fetched from git. In this case, set this to "none". +* **PACKAGE_DEPENDENCIES**: (Optional) If the build of this package requires + fetching artifacts from other linux-pkg packages, those should be specified + in PACKAGE_DEPENDENCIES, as a space-separated list. The dependencies will + be fetched in the `fetch_dependencies()` step into `//` where + "dep" is the dependency's name. A special value can be passed for packages + that target all the supported flavours of the linux-kernel: `@linux-kernel`. + * **DEFAULT_PACKAGE_GIT_BRANCH**: (DEPRECATED) Default git branch to use when fetching from or pushing to `DEFAULT_PACKAGE_GIT_URL`. This should be typically left unset. The branch to fetch the package from defaults @@ -355,88 +347,124 @@ Here is a list of variables that can be defined for a package: * **DEFAULT_PACKAGE_VERSION**: (Optional) The version of the package is set to this value when it is built. **Note:** If this field is not set, then you - should provide a mechanism in the [build](#build) hook to auto-determine the - version from the source code. + should provide a mechanism in the [build](#build-hook) hook to auto-determine + the version from the source code. + WARNING: This parameter will be removed in the near future, as we will rely on + the changelog contained in the package's repository to get the package version + in the future. * **DEFAULT_PACKAGE_REVISION**: (Optional) The revision of the package is set to this value when it is built (note that the full version of a package is "_VERSION-REVISION_"). If unset, it defaults to value of environment variable DEFAULT_REVISION. + WARNING: This parameter is currently unused and will be removed in the near + future. * **UPSTREAM_SOURCE_PACKAGE**: (Optional) Third-party packages that have an - [update_upstream](#update-upstream) hook and are updated from an Ubuntu source - package should set this to the name of the source package. + [update_upstream](#update-upstream-hook) hook and are updated from an Ubuntu + source package should set this to the name of the source package. * **UPSTREAM_GIT_URL, UPSTREAM_GIT_BRANCH**: (Optional) Third-party packages - that have an [update_upstream](#update-upstream) hook and are updated from a - git repository should set this to the upstream git url and branch. - -### Package hooks - -This is a list of hooks that can be defined for a package. Those are simply bash -functions that are called by `buildpkg.sh`. - -#### Prepare - -The `prepare()` hook is optional. It is called before calling the build hook and -normally installs the build dependencies for the package. - -#### Fetch + that have an [update_upstream](#update-upstream-hook) hook and are updated + from a git repository should set this to the upstream git url and branch. + +* **FORCE_PUSH_ON_UPDATE**: (Optional) This applies to some third-party packages + that have an [update_upstream](#update-upstream-hook) hook. Most third-party + packages are synced with upstream by performing a simple "git-merge" command, + so when the merge is pushed it can be done with "git push". However some + packages, like the linux-kernel ones, perform a rebase instead, and so the + merge must be force-pushed instead. If you want to use force push to push + an auto-merge, set FORCE_PUSH_ON_UPDATE to "true". Note that a safety check + is always performed prior to doing the push to make that the target branch + has not changed since the auto-merge commit was generated, however disabling + force-push by default is an extra precaution. + +* **SKIP_COPYRIGHTS_CHECK**: (Optional) By default, at the end of a package's + build we check that each produced deb contains a copyright file, unless + SKIP_COPYRIGHTS_CHECK is set to "true". + +### Package stages and hooks + +When operations are performed on a package by build or auto-update scripts, +such as [buildpkg.sh](#buildpkgsh) or +[sync-with-upstream.sh](#sync-with-upstreamsh), those operations are usually +split into high-level tasks called "stages". Some of those stages can be +modified or must be defined in a package's config file, so we refer to them +here as "hooks". Hooks that have a default definition are stored in +the `default-package-config.sh` file. + +Other "stages" are not meant to be modified and aren't functionally different +from regular function calls, we want to give them more visibility in the build +process as they are deemed as important high-levels tasks, so they are called +via the `stage()` helper function. + +#### Fetch (hook) The `fetch()` hook is optional, as a default is provided and should be used. It is called when fetching the source code of the package to build or to update. -The repository is cloned into `packages//tmp/repo` and checked out as +The repository is cloned into `/repo` and checked out as branch **repo-HEAD**. If we are performing a package update, then we also fetch the **upstreams/master** branch into **upstream-HEAD**. The default should only be overridden when not fetching the package source from git. -#### Build +#### Prepare (hook) + +The `prepare()` hook is optional. It is called before calling the build hook and +normally installs the build dependencies for the package. + +#### Build (hook) The `build()` hook is mandatory. It is responsible for building the package and storing the build products into `packages//tmp/artifacts/`. -#### Store Build Info +#### Update Upstream (hook) -The `store_build_info()` hook is optional. It is called right after the -build hook. It is responsible of creating the `/build_info` file that -contains information about the source of the code used to build the package. +The `update_upstream()` hook should only be defined for third party packages +that can be auto-updated. It is responsible for fetching the latest upstream +source code on top of branch **upstream-HEAD** of our fetched repository in +`/repo`. Note that any changes should be rebased on top of +the **upstreams/master** branch. If changes are detected, file +`/upstream-updated` should be created. -A default hook is provided in `default-package-config.sh` and will -be used if it is not overriden. If the package comes from a git repository, -the default hook will store the git hash, branch and repository of the -package's source. +#### Merge With Upstream (hook) -`build_info` files for each package are consumed by the -[metapackage](./build-info-pkg) when running [buildlist.sh](#buildlistsh). +The `merge_with_upstream()` hook is called after the `update_upstream()` hook +when a package is updated via [sync-with-upstream.sh](#sync-with-upstreamsh). +Whereas `update_upstream()` updates the **upstream-HEAD** branch, +`merge_with_upstream` then merges the **upstream-HEAD** branch into the +**repo-HEAD** branch. For most third-party packages this can be left unset as +the default will be used. For packages that have a more complex merge strategy, +such as the linux-kernel packages, this hook can be used. -#### Post Build Checks +#### Checkstyle (hook) -The `post_build_checks()` hook is optional. It is responsible for checking if -the debian package being built has copyright file associated with it in the -right location. This file will be used to generate the license information for -the appliance. +The `checkstyle()` hook is optional. It is called before building the package if +`-c` is provided to `buildpkg.sh`. Note that this hook isn't currently used by +our build automation and is more of a prototype for an idea. -A default hook is provided in `default-package-config.sh` and will -be used if it is not overriden. The packages that are eligible to skip this check -should define `SKIP_COPYRIGHTS_CHECK=true` in their respective `config.sh` files. +#### Fetch Dependencies -#### Checkstyle +`fetch_dependencies` is an immutable stage. It is called for fetching build +artifacts from other linux-pkg packages that are required for performing the +build. See the PACKAGE_DEPENDENCIES package variable for mroe info. -The `checkstyle()` hook is optional. It is called before building the package if -`-c` is provided to `buildpkg.sh`. +#### Store Build Info -#### Update Upstream +`store_build_info()` is an immutable stage. It is called after the `build()` +stage. It is responsible for storing some build info / metadata, such as the +git hash used to perform the build. Some of the build info that is stored is +used by build automation, so care must be exercised when modifying it. -The `update_upstream()` hook should only be defined for third party packages -that need to be auto-updated. It is responsible for fetching the latest upstream -source code on top of branch **upstream-HEAD** of our fetched repository in -`packages//tmp/repo`. Note that any changes should be rebased on top of -the **upstreams/master** branch. If changes are detected, file -`packages//tmp/upstream-updated` should be created. +#### Post Build Checks -After the `update_upstream()` hook is called, and if changes are detected, -`buildpkg.sh` will proceed to merge the **upstream-HEAD** branch into -**repo-HEAD** and build the resulting code. +`post_build_checks()` is an immutable stage. It is responsible for performing +post-build checks that are common to all packages. + +One of the checks verifies that each debian package produced has a copyright +file associated with it in the right location. This file is used elsewhere in +the product to generate the license information for the appliance. This check +can be skipped for a package by defining `SKIP_COPYRIGHTS_CHECK=true` in its +config file. ### Package environment variables @@ -461,7 +489,9 @@ variables are set-up by the framework. Here is a quick list: ### Package WORKDIR Each package is being fetched, built and updated in directory -`linux-pkg/packages//tmp/`, referred to as `WORKDIR`. +`linux-pkg/packages//tmp/`, referred to as `WORKDIR`. Whenever a +script is called to operate a package, the WORKDIR directory is recreated and +a `linux-pkg/workdir` symlink is created that points to this WORKDIR. The following sub-directories are created in `WORKDIR`: @@ -472,15 +502,14 @@ The following sub-directories are created in `WORKDIR`: * **source**: where the source package is fetched when updating upstream from a source package. -The following files are used as status indicators in `WORKDIR`: - -* **building**: created when package is being built, deleted on success. +The following files are created in `WORKDIR`: -* **updating-upstream**: created when updating upstream branch, deleted on - success. +* **upstream_tag**: During a package's auto-update, we may wish to also push + a tag fetched from the upstream repository for informational purposes. If so, + the `upstream_tag` file should be created and contain the name of the tag + that needs to be pushed. -* **merging**: created when package is being merged with upstream branch, - deleted on success. +The following files are used as status indicators in `WORKDIR`: * **upstream-updated**: created if **upstream-HEAD** has updates that should be pushed. @@ -488,18 +517,16 @@ The following files are used as status indicators in `WORKDIR`: * **repo-updated**: created if **repo-HEAD** has updates that should be pushed, following a merge. -* **build_info**: created by the `store_build_info()` package hook. - ## Adding new packages When considering adding a new package, the workflow will depend on whether the package is a [third-party package](#third-party-package) or [in-house package](#in-house-package). -**Note For Delphix Employees**: +**Note:**: If you are thinking of adding a new package to this framework, you should first read the -[Delphix Open-Source Policy](https://docs.delphix.com/cto/ip-strategy/outbound-open-source). +[Delphix Open-Source Policy](https://docs.delphix.com/en/ip-strategy/outbound-open-source). ### Third-party package @@ -551,14 +578,23 @@ branch that points to the **master** branch; you can then update `DEFAULT_PACKAGE_GIT_URL` in config.sh to your forked git repository and skip to step 6. -You can fetch the upstream source code by running: +You can fetch the upstream source code from an Ubuntu source package by running: ``` -./buildpkg.sh -i +cd packages//tmp/ +mkdir source +cd source +apt-get source +cd .. +mv source/""*/ repo +cd repo +git init +git checkout -b repo-HEAD +git add -f . +git commit -m '' ``` - -This will automatically fetch the code into `packages//tmp/repo` and -initialize it as a git repository. +TODO: create a command that will run the steps above. It used to be done by +`buildpkg.sh -i`, but this logic has been removed. #### Step 4. Create a developer repository @@ -573,20 +609,11 @@ e.g. DEFAULT_PACKAGE_GIT_URL="https://github.com//" ``` -Note that the branch will default to **master** unless -`DEFAULT_PACKAGE_GIT_BRANCH` is also provided. - #### Step 5. Push to your developer repository -Next step is to push the upstream code to the newly created repository using the -`push-update.sh` script. The script will need to be called twice: once for the -**upstreams/master** branch and once for the **master** branch. It will also -prompt you for your git credentials. - -``` -./push-updates -u -./push-updates -m -``` +Next step is to push the upstream code to the newly created repository to your +developer repository. You should push the initial commit to both the **master** +branch and the **upstreams/master** branch. #### Step 6. Build the package @@ -599,8 +626,10 @@ that will install those build dependencies. For an Ubuntu source package, those dependencies can be installed by calling `install_build_deps_from_control_file()`. For other packages, you can usually find the build dependencies in the project's -README. It is recommended to use the `install_pkgs()` function to install -packages. +README. It is recommended to edit the `debian/control` file of the package +to list the required build dependencies, so that +`install_build_deps_from_control_file()` can be used. Otherwise, you can also +use the `install_pkgs()` lib function to install packages. Next step is to add a [build()](#build) hook. It is recommended to use the `dpkg_buildpackage_default()` function. @@ -632,8 +661,8 @@ Once this is all ready, you can try building the package by running: #### Step 7. Make the package auto-updatable If you want the package to be automatically updated with upstream (strongly -recommended), you'll need to add the [update_upstream()](#update-upstream) hook -to `config.sh`. You should use the following functions provided by +recommended), you'll need to add the [update_upstream()](#update-upstream-hook) +hook to `config.sh`. You should use the following functions provided by [lib/common.sh](./lib/common.sh): * `update_upstream_from_source_package()` if `UPSTREAM_SOURCE_PACKAGE` is set. @@ -723,28 +752,15 @@ require a debian metadirectory. #### Add package to package-lists -* Add the new package to the appropriate build list in - [package-lists/build/](./package-lists/build/). - Most packages that will be deployed on the Delphix Appliance should be added - to the [userland.pkgs](./package-lists/build/userland.pkgs) list. - -* If this is a third-party package that is to be auto-updated by - `updatelist.sh`, it should also be added to - [package-lists/update/userland.pkgs](./package-lists/update/userland.pkgs). - -* To make sure that the new package is included in the Delphix Appliance by - appliance-build, it should be added as a dependency to an existing package - such as `delphix-platform` or `delphix-virtualization`. +See the [Package Lists](#package-lists) section for more info. #### Make the package official -**Note**: this step only applies to Delphix. - Once your new package builds and has been tested in the product, the next step is to create an official repository for it. 1. First, you should read - [Delphix Open-Source Policy](https://docs.delphix.com/cto/ip-strategy/outbound-open-source) + [Delphix Open-Source Policy](https://docs.delphix.com/en/ip-strategy/outbound-open-source) if you haven't already, and provide the necessary info so that a `github.com/delphix/` repository can be created for it. You'll need to push the **master** branch from your developer repository, as well as the @@ -768,34 +784,50 @@ package managed by linux-pkg: 1. Run `git-ab-pre-push` from your package's repository. -More instructions available -[here](https://docs.google.com/document/d/1pD0AusWAIbqXalx-B5nhrrHBfMme6wHvJG9c7O_wqb4/view). +TODO: complete section ### Testing changes to linux-pkg -If you are adding a new package, changing the linux-pkg framework, or changing -the build definition (config.sh file) of a package, you should perform the -following steps: +TODO: complete section + +## Package Lists -1. On your build VM, in the linux-pkg repository, run checkstyle: +Package lists are basically just lists of packages defined in linux-pkg. +They are mainly consumed by the Jenkins build infrastructure by calling +the [./query-packages.sh](./query-packages.sh) utility. Jenkins needs to know +which packages to build and include for a given version of the Delphix +appliance. - ``` - make clean - make check - ``` +Package lists are stored under [./package-lists](./package-lists), in two +sub-directories: `build` and `update`. The `build` directory contains packages +that are built and consumed by the Delphix Appliance, while the `update` +directory contains a list of packages that are automatically synced with +the upstream projects. -1. Run the Jenkins build jobs for the - [userland](http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/linux-pkg-build/job/master/job/kernel/job/pre-push/) - and - [kernel](http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/linux-pkg-build/job/master/job/userland/job/pre-push/) - build lists. +There are two physical `build` lists: -1. Run the Jenkins update job for the - [userland](http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/linux-pkg-update/job/master/job/userland/job/update/) - update list. +* `main.pkgs`: this is the default list for packages that are to be added to the + Delphix Appliance. -More instructions available -[here](https://docs.google.com/document/d/1pD0AusWAIbqXalx-B5nhrrHBfMme6wHvJG9c7O_wqb4/view). +* `kernel-modules.pkgs`: this list is similar to the `main` list but contains + packages that have a dependency on the multiple flavours of the linux kernel + that are supported by the Delphix Appliance. + +There's also a virtual build list, called "linux-kernel", which lists all the +linux kernel packages built by linux-pkg (one for each supported flavour of +the linux kernel). You can list the contents of the virtual list by running: + +``` +./query-packages.sh list linux-kernel +``` + +There is a single `update` list called `main.pkgs`, which contains all the +packages that are auto-updated nightly by Jenkins. Note that zfs is not in +that list as it has a dedicated Jenkins job that tracks the upstream +repository and launches as soon as there are new changes. + +Most third-party packages should have an `update_upstream()` hook defined and +be added to that list. ## Versions and Branches @@ -815,7 +847,7 @@ linux-pkg repository itself follows the Delphix branching policy outlined [here](https://docs.delphix.com/pages/viewpage.action?spaceKey=RE&title=New+Branching+Mechanism). When creating a new branch or release for the Delphix Appliance, an external script should create the relevant branch or tag for each repository. The -branch or tag should then be passed to the build in the `DEFAULT_BRANCH` +branch or tag should then be passed to the build in the `DEFAULT_GIT_BRANCH` environment variable. ### Future work @@ -825,15 +857,6 @@ image will need to be picked accordingly. We are currently using `bootstrap-18-04`, but this will not be the case anymore once we switch to a newer Ubuntu distribution. -Regarding auto-update of third-party packages, we'll most likely want to enable -support for other branches than master, especially _stage_ ones. This way we'd -be able to automatically pull in security updates for our third-party packages -that track Ubuntu source packages. - -This means that we will also need integration with our Ubuntu package mirrors. -The auto-update process will need to track the proper archive when fetching -source packages. - ## Contributing All contributors are required to sign the Delphix Contributor Agreement prior diff --git a/build-info-pkg/.gitignore b/build-info-pkg/.gitignore deleted file mode 100644 index 1e596e81..00000000 --- a/build-info-pkg/.gitignore +++ /dev/null @@ -1,8 +0,0 @@ -artifacts -debian/changelog -debian/control -debian/.debhelper -debian/delphix-buildinfo* -debian/files -debian/debhelper-build-stamp -lib \ No newline at end of file diff --git a/build-info-pkg/build-package.sh b/build-info-pkg/build-package.sh deleted file mode 100755 index 2ce114f6..00000000 --- a/build-info-pkg/build-package.sh +++ /dev/null @@ -1,89 +0,0 @@ -#!/bin/bash -e -# -# Copyright 2018, 2019 Delphix -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# shellcheck disable=SC2016 - -# -# This script is responsible of building the build-info package after a list -# of packages have been built. Each package may generate a build_info file -# containing custom metadata related to the build (e.g. see the store_git_info -# function in common.sh). This script concatenates the build_info files -# generated by each package into a single file that will be provided by this -# package: /lib/delphix-buildinfo/{pkg_list}.info. -# -# The name of the build-info package is "delphix-buildinfo-{pkg_list}" -# - -TOP="$(git rev-parse --show-toplevel)" -source "$TOP/lib/common.sh" - -SCRIPT_DIR="$TOP/build-info-pkg" - -[[ $# -eq 1 ]] || die "Must provide package list as argument." - -pkg_list="$1" -logmust get_package_list_file "build" "$pkg_list" -pkg_list_file="$_RET" - -logmust "$SCRIPT_DIR/clean.sh" - -logmust read_package_list "$pkg_list_file" -PACKAGES=("${_RET_LIST[@]}") - -INFO_FILE="$SCRIPT_DIR/lib/delphix-buildinfo/${pkg_list}.info" -logmust mkdir -p "$(dirname "$INFO_FILE")" - -LINUX_PKG_HASH="$(git rev-parse HEAD)" || die "git rev-parse HEAD failed" - -# -# LINUX_PKG_GIT_URL & LINUX_PKG_GIT_BRANCH are passed by Jenkins -# -cat <<-EOF >"$INFO_FILE" - Linux-pkg Package Framework: - Git hash: $LINUX_PKG_HASH - Git repo: ${LINUX_PKG_GIT_URL:-unknown} - Git branch: ${LINUX_PKG_GIT_BRANCH:-unknown} -EOF - -echo "" >>"$INFO_FILE" - -logmust cd "$TOP/packages" -for pkg in "${PACKAGES[@]}"; do - echo "Package $pkg:" - if [[ -f "${pkg}/tmp/build_info" ]]; then - cat "${pkg}/tmp/build_info" - else - echo "NO INFO" - fi - echo "" -done >>"$INFO_FILE" - -package_name="delphix-buildinfo-${pkg_list}" - -logmust cd "$SCRIPT_DIR" -logmust bash -c "sed 's/@@PACKAGE@@/$package_name/g' \ - debian/control.in >debian/control" - -export DEBEMAIL="Delphix Engineering " -export PACKAGE_VERSION=1.0.0 -export PACKAGE_REVISION=${DEFAULT_REVISION:-0} - -logmust set_changelog "$package_name" -logmust dpkg-buildpackage -uc -us -b -logmust mkdir -p artifacts -logmust mv ../"$package_name"*deb artifacts/ -logmust rm -f ../"$package_name"*.buildinfo ../"$package_name"*.changes -logmust cp "$INFO_FILE" artifacts/build-info diff --git a/build-info-pkg/debian/compat b/build-info-pkg/debian/compat deleted file mode 100644 index f599e28b..00000000 --- a/build-info-pkg/debian/compat +++ /dev/null @@ -1 +0,0 @@ -10 diff --git a/build-info-pkg/debian/copyright b/build-info-pkg/debian/copyright deleted file mode 100644 index 5bd31791..00000000 --- a/build-info-pkg/debian/copyright +++ /dev/null @@ -1,19 +0,0 @@ -Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ - -Files: * -Copyright: 2018 Delphix -License: Apache-2.0 - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - . - http://www.apache.org/licenses/LICENSE-2.0 - . - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - . - On Debian systems, the complete text of the Apache License, Version 2 - can be found in "/usr/share/common-licenses/Apache-2.0". diff --git a/build-info-pkg/debian/install b/build-info-pkg/debian/install deleted file mode 100644 index 502167fa..00000000 --- a/build-info-pkg/debian/install +++ /dev/null @@ -1 +0,0 @@ -/lib diff --git a/buildlist.sh b/buildlist.sh deleted file mode 100755 index e0b627bc..00000000 --- a/buildlist.sh +++ /dev/null @@ -1,87 +0,0 @@ -#!/bin/bash -# -# Copyright 2018, 2019 Delphix -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -# -# This script first builds a list of packages by running buildpkg.sh on each -# package, and then generates a build-info package. All the build products are -# stored in the ./artifacts directory. Valid package lists are stored in -# package-lists/build/ -# - -TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" -source "$TOP/lib/common.sh" - -logmust check_running_system - -function usage() { - [[ $# != 0 ]] && echo "$(basename "$0"): $*" - echo "Usage: $(basename "$0") " - echo "" - echo " This script fetches and builds all the packages defined in" - echo " package-lists/build/.pkgs." - echo "" - exit 2 -} - -[[ $# -eq 1 ]] || usage "takes exactly one argument." >&2 - -pkg_list="$1" -logmust get_package_list_file "build" "$pkg_list" -pkg_list_file="$_RET" - -logmust cd "$TOP" -logmust make clean -logmust mkdir artifacts - -# -# Auto-generate the default revision for all the packages. It will be the -# default used if the revision is not set explicitly anywhere else. -# -export DEFAULT_REVISION="${DEFAULT_REVISION:-$(default_revision)}" - -# -# A list of target platform or versions to build modules for can be passed in -# TARGET_PLATFORMS. Convert values like "aws" into actual kernel -# versions and store them into KERNEL_VERSIONS. -# -logmust determine_target_kernels -export KERNEL_VERSIONS - -build_flags="" -if [[ "$CHECKSTYLE" == "true" ]]; then - build_flags="${build_flags} -c" -fi - -# -# Get the list of packages to build. -# -logmust read_package_list "$pkg_list_file" -PACKAGES=("${_RET_LIST[@]}") - -for pkg in "${PACKAGES[@]}"; do - # shellcheck disable=SC2086 - logmust ./buildpkg.sh $build_flags "$pkg" -done - -logmust build-info-pkg/build-package.sh "$pkg_list" -logmust cp build-info-pkg/artifacts/* artifacts/ - -for pkg in "${PACKAGES[@]}"; do - logmust cp "packages/$pkg/tmp/artifacts"/* artifacts/ -done - -echo_success "Packages have been built successfully." diff --git a/buildpkg.sh b/buildpkg.sh index edd35605..04d9a54e 100755 --- a/buildpkg.sh +++ b/buildpkg.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018, 2019 Delphix +# Copyright 2018, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -20,97 +20,9 @@ source "$TOP/lib/common.sh" logmust check_running_system -function merge_with_upstream() { - local upstream_ref="refs/heads/upstream-HEAD" - - logmust cd "$WORKDIR/repo" - check_git_ref "$upstream_ref" "refs/heads/repo-HEAD" - - logmust git checkout -q repo-HEAD - - if git merge-base --is-ancestor "$upstream_ref" HEAD; then - echo "NOTE: $PACKAGE is already up-to-date with upstream." - return 0 - fi - - echo "Running: git merge --no-edit $upstream_ref" - if git merge --no-edit --no-stat "$upstream_ref"; then - echo "git merge succeeded" - logmust touch "$WORKDIR/repo-updated" - return 0 - else - echo "git merge failed" - logmust git merge --abort - return 1 - fi -} - -# -# Inititalize Delphix git repository from a source package. -# -function inititalize_from_upstream_source_package() { - check_env UPSTREAM_SOURCE_PACKAGE - - # - # Fetch the source package into source/ - # - logmust mkdir "$WORKDIR/source" - logmust cd "$WORKDIR/source" - logmust apt-get source "$UPSTREAM_SOURCE_PACKAGE" - - # - # Create initial repository from the package source. - # Both repo-HEAD and upstream-HEAD point to the same commit. - # - logmust cd "$WORKDIR" - logmust mv source/"$UPSTREAM_SOURCE_PACKAGE"*/ repo - logmust cd "$WORKDIR/repo" - logmust git init - logmust git checkout -b repo-HEAD - logmust git add -f . - logmust generate_commit_message_from_dsc - logmust git commit -F "$WORKDIR/commit-message" - logmust git branch upstream-HEAD - - logmust touch "$WORKDIR/upstream-updated" - logmust touch "$WORKDIR/repo-updated" -} - -# -# Inititalize Delphix git repository from an upstream git repository. -# -function inititalize_from_upstream_git() { - check_env UPSTREAM_GIT_URL UPSTREAM_GIT_BRANCH - - logmust mkdir "$WORKDIR/repo" - logmust cd "$WORKDIR/repo" - logmust git init - logmust git remote add upstream "$UPSTREAM_GIT_URL" - logmust git fetch upstream "$UPSTREAM_GIT_BRANCH" - - logmust git branch repo-HEAD FETCH_HEAD - logmust git branch upstream-HEAD FETCH_HEAD - - logmust git checkout -q repo-HEAD - - logmust touch "$WORKDIR/upstream-updated" - logmust touch "$WORKDIR/repo-updated" -} - -function inititalize_from_upstream() { - if [[ -n "$UPSTREAM_GIT_URL" ]]; then - logmust inititalize_from_upstream_git - elif [[ -n "$UPSTREAM_SOURCE_PACKAGE" ]]; then - logmust inititalize_from_upstream_source_package - else - die "$PACKAGE/config.sh must contain either" \ - "UPSTREAM_SOURCE_PACKAGE or UPSTREAM_GIT_URL/BRANCH." - fi -} - function usage() { [[ $# != 0 ]] && echo "$(basename "$0"): $*" - echo "Usage: $(basename "$0") [-i | -u [-M]] [-ch] [-g pkg_git_url]" + echo "Usage: $(basename "$0") [-ch] [-g pkg_git_url]" echo " [-b pkg_git_branch] [-v pkg_version] [-r pkg_revision]" echo " package" echo "" @@ -121,16 +33,9 @@ function usage() { echo " and then build it." echo " Options:" echo "" - echo " -i Create initial repo from an upstream git repo or" - echo " source package. Conflicts with -u." - echo " -u Update upstream branch and merge main branch with" - echo " upstream. Build only if main branch has changed." - echo " Conflicts with -i." - echo " -M When passed with -u, only update upstream branch and" - echo " never attempt to build." - echo " -c Call the checkstyle hook after fetching package." echo " -g override default git url for the package." echo " -b override default git branch for the package." + echo " -c also run package's checkstyle hook." echo " -v override default version for package." echo " -r override default revision for package." echo " -h display this message and exit." @@ -143,22 +48,14 @@ unset PARAM_PACKAGE_GIT_BRANCH unset PARAM_PACKAGE_VERSION unset PARAM_PACKAGE_REVISION -export DO_UPDATE_PACKAGE=false - do_checkstyle=false -do_initialize=false -do_merge=true -while getopts ':b:cg:hik:Mr:uv:' c; do +while getopts ':b:cg:hr:v:' c; do case "$c" in g) export PARAM_PACKAGE_GIT_URL="$OPTARG" ;; b) export PARAM_PACKAGE_GIT_BRANCH="$OPTARG" ;; v) export PARAM_PACKAGE_VERSION="$OPTARG" ;; r) export PARAM_PACKAGE_REVISION="$OPTARG" ;; - k) export TARGET_PLATFORMS="$OPTARG" ;; c) do_checkstyle=true ;; - i) do_initialize=true ;; - u) DO_UPDATE_PACKAGE=true ;; - M) do_merge=false ;; h) usage >&2 ;; *) usage "illegal option -- $OPTARG" >&2 ;; esac @@ -168,9 +65,6 @@ shift $((OPTIND - 1)) [[ $# -gt 1 ]] && usage "too many arguments" >&2 PACKAGE=$1 -$DO_UPDATE_PACKAGE && $do_initialize && usage "-i and -u are exclusive" >&2 -! $do_merge && ! $DO_UPDATE_PACKAGE && usage "-M requires -u" >&2 - logmust check_package_exists "$PACKAGE" # @@ -187,55 +81,25 @@ echo_bold "====================================================================" echo "" logmust load_package_config "$PACKAGE" - -logmust sudo rm -rf "$WORKDIR" -logmust mkdir "$WORKDIR" +logmust create_workdir logmust mkdir "$WORKDIR/artifacts" -if $do_initialize; then - logmust inititalize_from_upstream - echo_success "Repository initialized from upstream in $WORKDIR/repo" - exit 0 -fi - logmust cd "$WORKDIR" stage fetch -if $DO_UPDATE_PACKAGE; then - logmust cd "$WORKDIR" - logmust touch "$WORKDIR/updating-upstream" - type -t update_upstream >/dev/null || - die "$PACKAGE: Hook 'update_upstream()' not found!" - stage update_upstream - logmust rm "$WORKDIR/updating-upstream" - - if ! $do_merge; then - echo_bold "Not attempting to merge with upstream since" \ - "-M is set." - exit 0 - fi - - logmust touch "$WORKDIR/merging" - logmust merge_with_upstream - logmust rm "$WORKDIR/merging" - if [[ ! -f "$WORKDIR/repo-updated" ]]; then - echo_bold "Not building package $PACKAGE since we are doing" \ - "an update but the repo is already up-to-date" - exit 0 - fi -fi +logmust cd "$WORKDIR" +stage fetch_dependencies logmust cd "$WORKDIR" stage prepare -logmust touch "$WORKDIR/building" if $do_checkstyle; then logmust cd "$WORKDIR" stage checkstyle fi + logmust cd "$WORKDIR" stage build -logmust rm "$WORKDIR/building" logmust cd "$WORKDIR" stage store_build_info @@ -246,8 +110,3 @@ stage post_build_checks echo_success "Package $PACKAGE has been built successfully." echo "Build products are in $WORKDIR/artifacts" echo "" - -if $DO_UPDATE_PACKAGE; then - echo_success "Auto-merge with upstream performed" \ - "successfully in $WORKDIR/repo" -fi diff --git a/checkupdates.sh b/checkupdates.sh new file mode 100755 index 00000000..8ff0590d --- /dev/null +++ b/checkupdates.sh @@ -0,0 +1,80 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" +source "$TOP/lib/common.sh" + +logmust check_running_system + +function usage() { + [[ $# != 0 ]] && echo "$(basename "$0"): $*" + echo "Usage: $(basename "$0") " + echo "" + echo " This script checks if upstream updates are available for the" + echo " target package. It returns succesfully whether or not there are" + echo " updates available. If either the upstream can be updated or the" + echo " active branch can be merged with a previously updated upstream" + echo " then the file workdir/update-available will be created." + echo "" + echo " -h display this message and exit." + echo "" + exit 2 +} + +while getopts ':h' c; do + case "$c" in + h) usage >&2 ;; + *) usage "illegal option -- $OPTARG" >&2 ;; + esac +done +shift $((OPTIND - 1)) +[[ $# -lt 1 ]] && usage "package argument missing" >&2 +[[ $# -gt 1 ]] && usage "too many arguments" >&2 +PACKAGE=$1 + +logmust check_package_exists "$PACKAGE" + +# +# If the script is called manually, we set it here. +# +DEFAULT_REVISION="${DEFAULT_REVISION:-$(default_revision)}" +logmust determine_default_git_branch + +logmust load_package_config "$PACKAGE" +logmust create_workdir + +# +# Set DO_UPDATE_PACKAGE to true so that the fetch stage fetches both the +# target branch as well as the upstream branch. +# +export DO_UPDATE_PACKAGE=true +logmust cd "$WORKDIR" +stage fetch + +stage update_upstream +logmust is_merge_needed +merge_needed="$_RET" +$merge_needed && echo "Merge with upstream is needed." + +echo "" + +if [[ -f "$WORKDIR/upstream-updated" ]] || $merge_needed; then + logmust touch "$WORKDIR/update-available" + echo_success "Package $PACKAGE has updates available." +else + echo_bold "Package $PACKAGE is already up-to-date." +fi diff --git a/default-package-config.sh b/default-package-config.sh index 442d3257..3a950723 100644 --- a/default-package-config.sh +++ b/default-package-config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -27,12 +27,8 @@ function fetch() { logmust fetch_repo_from_git } -function store_build_info() { - if [[ -d "$WORKDIR/repo/.git" ]]; then - logmust store_git_info - else - echo "No build info available" >"$WORKDIR/build_info" - fi +function merge_with_upstream() { + logmust merge_with_upstream_default } # @@ -93,9 +89,17 @@ function kernel_build() { # the end of the new one to maintain the mapping between # Canonical's releases and our releases. # - local canonical_abinum delphix_abinum - canonical_abinum=$(fakeroot debian/rules printenv | grep abinum | cut -d= -f2 | tr -d '[:space:]') + local canonical_abinum delphix_abinum kernel_release kernel_version + canonical_abinum=$(fakeroot debian/rules printenv | grep -E '^abinum ' | cut -d= -f2 | tr -d '[:space:]') delphix_abinum="dlpx-$(date -u +"%Y%m%dt%H%M%S")-$(git rev-parse --short HEAD)-${canonical_abinum}" + kernel_release=$(fakeroot debian/rules printenv | grep -E '^release ' | cut -d= -f2 | tr -d '[:space:]') + + # + # We record the kernel version into a file. This field is consumed + # by other kernel packages, such as zfs, during their build. + # + kernel_version="${kernel_release}-${delphix_abinum}-${platform}" + echo "$kernel_version" >"$WORKDIR/artifacts/KERNEL_VERSION" # # skipdbg=false @@ -106,8 +110,11 @@ function kernel_build() { # any intention and logic to provide signatures for now # we set it to false to avoid any misconfigurations down # the line. + # disable_d_i=true + # This prevents udeb packages from being built as they are + # not consumed by the Delphix Appliance. # - local debian_rules_args="skipdbg=false uefi_signed=false abinum=${delphix_abinum} ${debian_rules_extra_args}" + local debian_rules_args="skipdbg=false uefi_signed=false disable_d_i=true flavours=$platform abinum=${delphix_abinum} ${debian_rules_extra_args}" # # Clean up everything generated so far and recreate the @@ -132,10 +139,16 @@ function kernel_build() { local build_deps_tool="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends --yes" logmust sudo mk-build-deps --install debian/control --tool "${build_deps_tool}" - logmust fakeroot debian/rules "binary-${platform}" ${debian_rules_args} + logmust fakeroot debian/rules "binary" ${debian_rules_args} logmust cd "$WORKDIR" logmust mv ./*deb "artifacts/" + + # + # Make sure that we recorded the kernel version properly by checking + # one of the .debs produced + # + logmust test -f "artifacts/linux-image-${kernel_version}_"*.deb } # @@ -195,7 +208,7 @@ function kernel_update_upstream() { # out the latest upstream tag to sync with. # local kernel_version abinum - logmust get_kernel_for_platform "${platform}" + logmust get_kernel_version_for_platform_from_apt "${platform}" kernel_version=$(echo "$_RET" | cut -d '-' -f 1) abinum=$(echo "$_RET" | cut -d '-' -f 2) @@ -258,8 +271,60 @@ function kernel_update_upstream() { local upstream_tag upstream_tag=$(echo "${upstream_tag_info}" | awk -F / '{print $3}') [[ -z "${upstream_tag}" ]] && die "could not extract upstream tag name from the tag info" - echo "note: upstream tag: ${upstream_tag}" - logmust git fetch upstream "${upstream_tag}" + + logmust git fetch upstream "+refs/tags/${upstream_tag}:refs/tags/${upstream_tag}" + + local upstream_tag_commit + upstream_tag_commit="$(git rev-parse "refs/tags/${upstream_tag}")" || + die "couldn't get commit of tag ${upstream_tag}" + echo "note: upstream tag: ${upstream_tag}, commit ${upstream_tag_commit}" + + # + # Check if the commit of the latest tag from upstream matches + # what we have cached in our repository at upstreams/, + # which we fetch to upstream-HEAD. + # + local local_upstream_commit + local_upstream_commit=$(git rev-parse upstream-HEAD) + [[ -z "${local_upstream_commit}" ]] && die "could not find upstream-HEAD's commit" + echo "note: upstreams/${DEFAULT_GIT_BRANCH} commit: ${local_upstream_commit}" + + if [[ "${upstream_tag_commit}" == "${local_upstream_commit}" ]]; then + echo "NOTE: upstream for $PACKAGE is already up-to-date." + else + logmust git reset --hard "refs/tags/${upstream_tag}" + echo "NOTE: upstream updated to refs/tags/${upstream_tag}" + + # + # Store name of upstream tag so that we can push it to our + # repository for reference purposes. + # + echo "refs/tags/${upstream_tag}" >"$WORKDIR/upstream-tag" || + die "failed to write to $WORKDIR/upstream-tag" + + logmust touch "$WORKDIR/upstream-updated" + fi + + logmust cd "$WORKDIR" +} + +# +# This merges local changes in repo-HEAD with upstream changes in upstream-HEAD. +# As opposed to the default merge function merge_with_upstream_default(), this +# uses git cherry-pick to rebase our changes on top of the upstream changes. +# +function kernel_merge_with_upstream() { + local repo_ref="refs/heads/repo-HEAD" + local upstream_ref="refs/heads/upstream-HEAD" + + logmust cd "$WORKDIR/repo" + + check_git_ref "$upstream_ref" "$repo_ref" + + if git merge-base --is-ancestor "$upstream_ref" "$repo_ref"; then + echo "NOTE: $PACKAGE is already up-to-date with upstream." + return 0 + fi # # Ensure that there is a commit marking the start of @@ -276,29 +341,20 @@ function kernel_update_upstream() { [[ -z "${dlpx_patch_end}" ]] && die "could not find repo-HEAD's head commit" # - # Compare that commit with the head commit of the - # upstream tag. If the commits are the same then - # there is nothing for us to do as we are using - # the most up-to-date tag as the base for our set - # of patches. On the other hand, if the commits - # differ then it means that the upstream has been - # updated, at which point we need to cherry-pick - # our patches on top of the new upstream. + # We rebase all the Delphix commits on top of the new upstream-HEAD + # by using git cherry-pick. Note that we also save the previous + # tip of the active branch to repo-HEAD-saved as this reference will be + # checked later by push-merge.sh. # - local upstream_head_commit - upstream_head_commit=$(git rev-parse upstream-HEAD) - [[ -z "${upstream_head_commit}" ]] && die "could not find upstream-HEAD's head commit" - if [[ "${current_ubuntu_commit}" == "${upstream_head_commit}" ]]; then - echo "NOTE: upstream for $PACKAGE is already up-to-date." - else - # shellcheck disable=SC2086 - logmust git cherry-pick ${dlpx_patch_start}^..${dlpx_patch_end} + logmust git branch repo-HEAD-saved repo-HEAD + logmust git branch -D repo-HEAD + logmust git checkout -q -b repo-HEAD upstream-HEAD - logmust touch "$WORKDIR/upstream-updated" - fi + # shellcheck disable=SC2086 + logmust git cherry-pick ${dlpx_patch_start}^..${dlpx_patch_end} - logmust cd "$WORKDIR" + logmust touch "$WORKDIR/repo-updated" } function post_build_checks() { diff --git a/jenkins-buildlist.sh b/jenkins-buildlist.sh deleted file mode 100755 index dfcd72dd..00000000 --- a/jenkins-buildlist.sh +++ /dev/null @@ -1,151 +0,0 @@ -#!/bin/bash -# -# Copyright 2018, 2019 Delphix -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" -source "$TOP/lib/common.sh" - -logmust check_running_system - -function usage() { - [[ $# != 0 ]] && echo "$(basename "$0"): $*" - echo "Usage: $(basename "$0")" - echo "" - echo " This is a wrapper script that is meant to be called from" - echo " Jenkins. It consumes and processes environment variables" - echo " passed from Jenkins and call 'buildlist.sh '." - echo "" - exit 2 -} - -# -# BUILDER_CUSTOM_ENV is meant to be used to pass GIT_URL, GIT_BRANCH, VERSION -# and REVISION values for specific packages, by setting _ -# variables. See get_package_config_from_env() in lib/common.sh for more info. -# -# e.g.: -# CLOUD_INIT_GIT_BRANCH=feature-branch-1 -# CONNSTAT_GIT_URL=github.com/connstat-developer/connstat.git -# -function parse_custom_env() { - local allowed_vars=() - local prefix - local pkg - - # - # Build a list of allowed custom environment variables. - # - for pkg in "${PACKAGES[@]}"; do - get_package_prefix "$pkg" - prefix="$_RET" - for suffix in GIT_URL GIT_BRANCH VERSION REVISION; do - allowed_vars+=("${prefix}_${suffix}") - done - done - - # - # Parse each line in the custom env and check if it matches any of - # the allowed variables. - # - local found - while IFS= read -r line; do - # trim whitespace - line=$(echo "$line" | sed 's/^\s*//;s/\s*$//') - [[ -z "$line" ]] && continue - - if [[ "$line" =~ ([^=]+)=.* ]]; then - var="${BASH_REMATCH[1]}" - found=false - for allowed_var in "${allowed_vars[@]}"; do - if [[ "$allowed_var" == "$var" ]]; then - found=true - break - fi - done - $found || die "Parsing BUILDER_CUSTOM_ENV: '$var'" \ - "is not an allowed environment variable." - logmust export "$line" - else - die "Parsing BUILDER_CUSTOM_ENV: invalid entry '$line'" - fi - done < <(printf '%s\n' "$BUILDER_CUSTOM_ENV") -} - -[[ $# -eq 0 ]] || usage "takes no arguments." >&2 - -# -# Validate the list of packages to build. -# -check_env BUILD_LIST -logmust get_package_list_file "build" "$BUILD_LIST" -pkg_list_file="$_RET" -logmust read_package_list "$pkg_list_file" -PACKAGES=("${_RET_LIST[@]}") - -if [[ -n "$BUILDER_CUSTOM_ENV" ]]; then - logmust parse_custom_env -fi - -if [[ -n "$SINGLE_PACKAGE_NAME" ]]; then - logmust check_package_exists "$SINGLE_PACKAGE_NAME" - # - # Make sure that the package is actually part of the BUILD_LIST. - # - found=false - for pkg in "${PACKAGES[@]}"; do - if [[ "$pkg" == "$SINGLE_PACKAGE_NAME" ]]; then - found=true - break - fi - done - $found || die "Package SINGLE_PACKAGE_NAME=$SINGLE_PACKAGE_NAME is not" \ - "in package list '$BUILD_LIST'" - - # - # The following env parameters are propagated from jenkins: - # - # SINGLE_PACKAGE_GIT_URL, SINGLE_PACKAGE_GIT_BRANCH, - # SINGLE_PACKAGE_VERSION, SINGLE_PACKAGE_REVISION - # - # We make sure those variables are applied to package - # SINGLE_PACKAGE_NAME by copying values of SINGLE_PACKAGE_ into - # _. See comment for parse_custom_env() above. - # - logmust get_package_prefix "$SINGLE_PACKAGE_NAME" - prefix="$_RET" - echo_bold "Setting ${prefix}_ variables since" \ - "SINGLE_PACKAGE_NAME=$SINGLE_PACKAGE_NAME ..." - if [[ -n "$SINGLE_PACKAGE_GIT_URL" ]]; then - var="${prefix}_GIT_URL" - logmust export "${var}=$SINGLE_PACKAGE_GIT_URL" - fi - if [[ -n "$SINGLE_PACKAGE_GIT_BRANCH" ]]; then - var="${prefix}_GIT_BRANCH" - logmust export "${var}=$SINGLE_PACKAGE_GIT_BRANCH" - fi - if [[ -n "$SINGLE_PACKAGE_VERSION" ]]; then - var="${prefix}_VERSION" - logmust export "${var}=$SINGLE_PACKAGE_VERSION" - fi - if [[ -n "$SINGLE_PACKAGE_REVISION" ]]; then - var="${prefix}_REVISION" - logmust export "${var}=$SINGLE_PACKAGE_REVISION" - fi -fi - -logmust cd "$TOP" -logmust ./setup.sh -logmust ./buildlist.sh "$BUILD_LIST" diff --git a/jenkins-updatelist.sh b/jenkins-updatelist.sh deleted file mode 100755 index 49fa4b2b..00000000 --- a/jenkins-updatelist.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash -# -# Copyright 2019 Delphix -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" -source "$TOP/lib/common.sh" - -logmust check_running_system - -function usage() { - [[ $# != 0 ]] && echo "$(basename "$0"): $*" - echo "Usage: $(basename "$0")" - echo "" - echo " This is a wrapper script that is meant to be called from" - echo " Jenkins. It consumes and processes environment variables" - echo " passed from Jenkins and call 'updatelist.sh '." - echo "" - exit 2 -} - -[[ $# -eq 0 ]] || usage "takes no arguments." >&2 - -# -# Validate the list of packages to update and make sure the GIT_DRY_RUN -# environment variable is passed. -# -check_env UPDATE_LIST GIT_DRY_RUN -logmust get_package_list_file "update" "$UPDATE_LIST" - -if [[ "$GIT_DRY_RUN" == "false" ]]; then - dry_run='' -else - dry_run='-n' -fi - -logmust cd "$TOP" -logmust ./setup.sh -logmust ./updatelist.sh $dry_run "$UPDATE_LIST" diff --git a/lib/common.sh b/lib/common.sh index 8cc6658a..f181ea29 100644 --- a/lib/common.sh +++ b/lib/common.sh @@ -19,12 +19,25 @@ export _RET export _RET_LIST export DEBIAN_FRONTEND=noninteractive -# TODO: allow updating upstream for other branches than master -export REPO_UPSTREAM_BRANCH="upstreams/master" export SUPPORTED_KERNEL_FLAVORS="generic aws gcp azure oracle" +# +# Used when fetching artifacts for external dependencies. Can be overridden +# for testing purposes to use jenkins-ops. instead. +# +export JENKINS_OPS_DIR="${JENKINS_OPS_DIR:-jenkins-ops}" +export _BASE_S3_URL="s3://snapshot-de-images/builds/${JENKINS_OPS_DIR}/devops-gate/master" + export UBUNTU_DISTRIBUTION="bionic" +# +# We currently support getting the linux kernel from 3 different sources: +# 1. "delphix": building it from code +# 2. "archive": dowloading from apt +# 3. "prebuilt": pre-built kernel stored in artifactory +# +export DEFAULT_LINUX_KERNEL_PACKAGE_SOURCE="archive" + # shellcheck disable=SC2086 function enable_colors() { [[ -t 1 ]] && flags="" || flags="-T xterm" @@ -165,25 +178,6 @@ function check_git_ref() { done } -function query_git_credentials() { - if [[ -n "$PUSH_GIT_USER" ]] && [[ -n "$PUSH_GIT_PASSWORD" ]]; then - return 0 - fi - - if [[ ! -t 1 ]]; then - die "PUSH_GIT_USER and PUSH_GIT_PASSWORD environment" \ - "variables must be set to a user that has" \ - "push permissions for the target repository." - fi - - echo "Please enter git credentials for pushing to repository." - read -r -p "User: " PUSH_GIT_USER - read -r -s -p "Password: " PUSH_GIT_PASSWORD - echo "" - export PUSH_GIT_USER - export PUSH_GIT_PASSWORD -} - # # execute a hook from a package's config.sh # @@ -209,7 +203,16 @@ function reset_package_config_variables() { local hook local var - for hook in prepare fetch build checkstyle update_upstream; do + local hooks=" + prepare + fetch + build + checkstyle + update_upstream + merge_with_upstream + " + + for hook in $hooks; do unset "$hook" done @@ -229,6 +232,7 @@ function reset_package_config_variables() { WORKDIR PKGDIR PACKAGE_PREFIX + FORCE_PUSH_ON_UPDATE SKIP_COPYRIGHTS_CHECK " @@ -446,6 +450,17 @@ function get_package_config_from_env() { echo_bold "------------------------------------------------------------" } +function create_workdir() { + check_env WORKDIR + logmust sudo rm -rf "$WORKDIR" + logmust mkdir "$WORKDIR" + logmust rm -f "$TOP/workdir" + logmust ln -s "$WORKDIR" "$TOP/workdir" +} + +# +# apt install packages. +# function install_pkgs() { for attempt in {1..3}; do echo "Running: sudo env DEBIAN_FRONTEND=noninteractive " \ @@ -458,6 +473,9 @@ function install_pkgs() { die "apt-get install failed after $attempt attempts" } +# +# Install build dependencies listed in the debian/control file of the package. +# function install_build_deps_from_control_file() { logmust pushd "$WORKDIR/repo" logmust sudo env DEBIAN_FRONTEND=noninteractive mk-build-deps --install \ @@ -466,6 +484,9 @@ function install_build_deps_from_control_file() { logmust popd } +# +# Returns a list of all known packages in _RET_LIST. +# function list_all_packages() { local pkg @@ -479,6 +500,9 @@ function list_all_packages() { done } +# +# Read a package-list file and return listed packages in _RET_LIST. +# function read_package_list() { local file="$1" @@ -500,28 +524,6 @@ function read_package_list() { done <"$file" || die "Failed to read package list: $file" } -function get_package_list_file() { - local list_type="$1" - local list_name="$2" - - if [[ "$list_type" != build ]] && [[ "$list_type" != update ]]; then - die "Invalid list type '$list_type'" - fi - - _RET="$TOP/package-lists/${list_type}/${list_name}.pkgs" - if [[ ! -f "$_RET" ]]; then - echo_error "Invalid $list_type package list '$list_name'" - echo_error "See lists in $TOP/package-lists/${list_type}/." - echo_error "Choose one of:" - cd "$TOP/package-lists/${list_type}/" || - die "failed to cd to $TOP/package-lists/${list_type}/" - for list in *.pkgs; do - echo_error " ${list%.pkgs}" - done - die - fi -} - # # List all target kernel packages. By default, it returns all the kernel # flavors supported and built by linux-pkg, however this can be overridden @@ -544,7 +546,6 @@ function list_linux_kernel_packages() { _RET_LIST+=("linux-kernel-$kernel") done fi - return 0 } @@ -557,19 +558,36 @@ function install_shfmt() { echo "shfmt version $(shfmt -version) is installed." } +# +# Install kernel headers packages for all target kernels. +# The kernel packages are fetched from S3. +# function install_kernel_headers() { logmust determine_target_kernels - check_env KERNEL_VERSIONS + check_env KERNEL_VERSIONS DEPDIR - local kernel - local headers_pkgs="" + logmust list_linux_kernel_packages + # Note: linux packages returned in _RET_LIST - for kernel in $KERNEL_VERSIONS; do - headers_pkgs="$headers_pkgs linux-headers-$kernel" + # + # On some platforms there are 2 headers packages and both must be + # installed. Here's an example on AWS: + # - linux-headers-5.3.0-1030-aws_5.3.0-1030.32~18.04.1_amd64.deb + # - linux-aws-5.3-headers-5.3.0-1030_5.3.0-1030.32~18.04.1_all.deb + # + local pkg + for pkg in "${_RET_LIST[@]}"; do + logmust install_pkgs "$DEPDIR/$pkg/"*-headers-*.deb done - # shellcheck disable=SC2086 - logmust install_pkgs $headers_pkgs + # + # Verify that headers are installed for all kernel versions + # stored in KERNEL_VERSIONS + # + local kernel + for kernel in $KERNEL_VERSIONS; do + logmust dpkg-query -l "linux-headers-$kernel" >/dev/null + done } function default_revision() { @@ -590,11 +608,58 @@ function default_revision() { echo "delphix-$(date '+%Y.%m.%d.%H')" } +# +# Fetch artifacts from S3 for all packages listed in PACKAGE_DEPENDENCIES which +# is defined in the package's config. +# +function fetch_dependencies() { + export DEPDIR="$WORKDIR/dependencies" + logmust mkdir "$DEPDIR" + logmust cd "$DEPDIR" + + if [[ -z "$PACKAGE_DEPENDENCIES" ]]; then + echo "Package has no linux-pkg dependencies to fetch." + return + fi + + local base_url="$_BASE_S3_URL/linux-pkg/$DEFAULT_GIT_BRANCH/build-package" + + local bucket="${_BASE_S3_URL#s3://}" + bucket=${bucket%%/*} + + local dep s3urlvar s3url + for dep in $PACKAGE_DEPENDENCIES; do + echo "Fetching artifacts for dependency '$dep' ..." + get_package_prefix "$dep" + s3urlvar="${_RET}_S3_URL" + if [[ -n "${!s3urlvar}" ]]; then + s3url="${!s3urlvar}" + echo "S3 URL of package dependency '$dep' provided" \ + "externally" + echo "$s3urlvar=$s3url" + else + s3url="$base_url/$dep/post-push" + (logmust aws s3 cp --only-show-errors "$s3url/latest" .) || + die "Artifacts for dependency '$dep' missing." \ + "Dependency must be built first." + logmust cat latest + s3url="s3://$bucket/$(cat latest)" + logmust rm latest + fi + [[ "$s3url" != */ ]] && s3url="$s3url/" + logmust mkdir "$dep" + logmust aws s3 ls "$s3url" + logmust aws s3 cp --only-show-errors --recursive "$s3url" "$dep/" + echo_bold "Fetched artifacts for '$dep' from $s3url" + PACKAGE_DEPENDENCIES_METADATA="${PACKAGE_DEPENDENCIES_METADATA}$dep: $s3url\\n" + done +} + # # Fetch package repository into $WORKDIR/repo # function fetch_repo_from_git() { - check_env PACKAGE_GIT_URL PACKAGE_GIT_BRANCH + check_env PACKAGE_GIT_URL PACKAGE_GIT_BRANCH DEFAULT_GIT_BRANCH logmust mkdir "$WORKDIR/repo" logmust cd "$WORKDIR/repo" @@ -605,15 +670,17 @@ function fetch_repo_from_git() { # main branch and the upstream branch with their histories. # Otherwise just get the latest commit of the main branch. # - if $DO_UPDATE_PACKAGE; then - check_env REPO_UPSTREAM_BRANCH + if [[ "$DO_UPDATE_PACKAGE" == "true" ]]; then logmust git fetch --no-tags "$PACKAGE_GIT_URL" \ "+$PACKAGE_GIT_BRANCH:repo-HEAD" logmust git fetch --no-tags "$PACKAGE_GIT_URL" \ - "+$REPO_UPSTREAM_BRANCH:upstream-HEAD" + "+upstreams/$DEFAULT_GIT_BRANCH:upstream-HEAD" + logmust git show-ref repo-HEAD + logmust git show-ref upstream-HEAD else logmust git fetch --no-tags "$PACKAGE_GIT_URL" \ "+$PACKAGE_GIT_BRANCH:repo-HEAD" --depth=1 + logmust git show-ref repo-HEAD fi logmust git checkout repo-HEAD @@ -631,6 +698,11 @@ function generate_commit_message_from_dsc() { cat "$dsc" >>"$WORKDIR/commit-message" } +# +# Fetches branch upstreams/ from our default repository into local +# upstream-HEAD branch, then attempts to update it with changes from +# the source package UPSTREAM_SOURCE_PACKAGE, fetched from apt. +# function update_upstream_from_source_package() { check_env PACKAGE_GIT_BRANCH UPSTREAM_SOURCE_PACKAGE @@ -669,6 +741,7 @@ function update_upstream_from_source_package() { else logmust generate_commit_message_from_dsc logmust git commit -F "$WORKDIR/commit-message" + logmust git show-ref upstream-HEAD logmust touch "$WORKDIR/upstream-updated" fi @@ -676,6 +749,11 @@ function update_upstream_from_source_package() { logmust cd "$WORKDIR" } +# +# Fetches branch upstreams/ from our default repository into local +# upstream-HEAD branch, then attempts to update it with changes from +# the remote repository specified by UPSTREAM_GIT_URL and UPSTREAM_GIT_BRANCH. +# function update_upstream_from_git() { check_env UPSTREAM_GIT_URL UPSTREAM_GIT_BRANCH logmust cd "$WORKDIR/repo" @@ -705,6 +783,7 @@ function update_upstream_from_git() { # then we definitely want to be notified. # logmust git merge --no-edit --ff-only --no-stat FETCH_HEAD + logmust git show-ref upstream-HEAD logmust touch "$WORKDIR/upstream-updated" fi @@ -712,6 +791,115 @@ function update_upstream_from_git() { logmust cd "$WORKDIR" } +# +# Returns true if upstreams/ needs to be merged into for the +# active package, where is the branch being updated, i.e. +# DEFAULT_GIT_BRANCH. +# +function is_merge_needed() { + local repo_ref="refs/heads/repo-HEAD" + local upstream_ref="refs/heads/upstream-HEAD" + + logmust pushd "$WORKDIR/repo" + check_git_ref "$upstream_ref" "$repo_ref" + + if git merge-base --is-ancestor "$upstream_ref" "$repo_ref"; then + echo "Upstream is already merged into repo-HEAD" + _RET=false + else + _RET=true + fi + logmust popd +} + +# +# Default function for merging upstreams/ into , where +# is the branch being updated, i.e. DEFAULT_GIT_BRANCH. Note that this function +# does not actually look at the upstream repository itself, but relies on +# local branches repo-HEAD and upstream-HEAD to be present. +# +# If merge was needed, file $WORKDIR/repo-updated is created and previous tip +# of is saved in repo-HEAD-saved. The repo-updated file lets the +# caller (typically Jenkins) know if a merge was necessary. The repo-HEAD-saved +# ref should be compared to the remote branch when it is time to push the +# merge; if they differ it means that the remote branch was modified and +# so the merge should be aborted -- this can happen if a PR was merged by a +# developer while auto-update was running. +# +function merge_with_upstream_default() { + local repo_ref="refs/heads/repo-HEAD" + local upstream_ref="refs/heads/upstream-HEAD" + + logmust cd "$WORKDIR/repo" + check_git_ref "$upstream_ref" "$repo_ref" + + logmust git checkout -q repo-HEAD + + if git merge-base --is-ancestor "$upstream_ref" HEAD; then + echo "NOTE: $PACKAGE is already up-to-date with upstream." + return 0 + fi + + # + # Do a backup of the repo-HEAD branch so that it can be compared to the + # remote when time comes to do a push. + # + logmust git branch repo-HEAD-saved + + logmust git merge --no-edit --no-stat "$upstream_ref" + logmust git show-ref repo-HEAD + logmust touch "$WORKDIR/repo-updated" +} + +# +# Check if git credentials are set for pushing update. If running in +# interactive mode, it will prompt the user for credentials if they are not +# provided in env. +# +function check_git_credentials_set() { + if [[ -z "$PUSH_GIT_USER" ]] || [[ -z "$PUSH_GIT_PASSWORD" ]]; then + if [[ -t 1 ]]; then + if [[ "$DRYRUN" == "false" ]]; then + echo_bold "WARNING: this is NOT a dry-run, you are pushing to" \ + "a production branch" + fi + echo "Please enter git credentials to push to remote ($DEFAULT_PACKAGE_GIT_URL)." + read -r -p "Username: " PUSH_GIT_USER + read -r -s -p "Password: " PUSH_GIT_PASSWORD + export PUSH_GIT_USER + export PUSH_GIT_PASSWORD + else + die "PUSH_GIT_USER and PUSH_GIT_PASSWORD must be set." + fi + fi +} + +# +# Push a local ref to a remote ref of the default remote repository for the +# package. +# +function push_to_remote() { + local local_ref="$1" + local remote_ref="$2" + local force="${3:-false}" + + local flags="" + $force && flags="-f" + + logmust check_git_credentials_set + + check_env DEFAULT_PACKAGE_GIT_URL PUSH_GIT_USER PUSH_GIT_PASSWORD + local git_url_with_creds="${DEFAULT_PACKAGE_GIT_URL/https:\/\//https:\/\/${PUSH_GIT_USER}:${PUSH_GIT_PASSWORD}@}" + local git_url_with_fake_creds="${DEFAULT_PACKAGE_GIT_URL/https:\/\//https:\/\/${PUSH_GIT_USER}:@}" + + logmust cd "$WORKDIR/repo" + check_git_ref "$local_ref" + + echo "RUNNING: git push $flags $git_url_with_fake_creds $local_ref:$remote_ref" + git push $flags "$git_url_with_creds" "$local_ref:$remote_ref" || + die "push failed" +} + # # Creates a new changelog entry for the package with the appropriate fields. # If no changelog file exists, source package name can be passed in first arg. @@ -733,6 +921,10 @@ function set_changelog() { fi } +# +# Default dpkg_buildpackage function for building packages. Before running the +# build, it updates the version of the package in the changelog. +# function dpkg_buildpackage_default() { logmust cd "$WORKDIR/repo" logmust set_changelog @@ -741,25 +933,11 @@ function dpkg_buildpackage_default() { logmust mv ./*deb artifacts/ } -# -# Store some metadata about what was this package built from. When running -# buildlist.sh, build_info for all packages is ingested by the metapackage -# and installed into /lib/delphix-buildinfo/.info. -# -function store_git_info() { - logmust pushd "$WORKDIR/repo" - echo "Git hash: $(git rev-parse HEAD)" >"$WORKDIR/build_info" || - die "storing git info failed" - echo "Git repo: $PACKAGE_GIT_URL" >>"$WORKDIR/build_info" - echo "Git branch: $PACKAGE_GIT_BRANCH" >>"$WORKDIR/build_info" - logmust popd -} - # # Returns the default (usually latest) kernel version for a given platform. # Result is placed into _RET. # -function get_kernel_for_platform() { +function get_kernel_version_for_platform_from_apt() { local platform="$1" local package @@ -796,52 +974,114 @@ function get_kernel_for_platform() { fi } +# +# Given a kernel version, fetch all necessary linux kernel packages +# into WORKDIR/artifacts. Also store kernel version into KERNEL_VERSION. +# +function fetch_kernel_from_apt_for_version() { + local kernel_version="$1" + + logmust cd "$WORKDIR/artifacts" + logmust apt-get download \ + "linux-image-${kernel_version}" \ + "linux-image-${kernel_version}-dbgsym" \ + "linux-modules-${kernel_version}" \ + "linux-headers-${kernel_version}" \ + "linux-tools-${kernel_version}" + + # + # Fetch direct dependencies of the downloaded debs. Some of those + # dependencies have a slightly different naming scheme than the other + # kernel packages. + # + local deb dep deps + for deb in *.deb; do + deps=$(dpkg-deb -f "$deb" Depends | tr -d ' ' | tr ',' ' ') || + die "failed to get dependencies for $deb" + for dep in $deps; do + case "$dep" in + *-headers-* | *-tools-*) + logmust apt-get download "$dep" + ;; + esac + done + done + + echo "$kernel_version" >KERNEL_VERSION +} + +# +# Find latest linux kernel available in apt for the given platform, and +# download all the necessary linux-kernel packages. +# +function fetch_kernel_from_apt_for_platform() { + local platform="$1" + + local kernel_version + logmust get_kernel_version_for_platform_from_apt "$platform" + kernel_version="$_RET" + + logmust fetch_kernel_from_apt_for_version "$kernel_version" +} + +# +# Fetch linux kernel packages from apt for the given kernel version. Also +# fetch the pre-built linux-modules package from artifactory. The pre-built +# package should have the same name as the one downloaded from apt but +# a higher revision number so that it will be picked over the default one +# downloaded from apt during the build of the appliance. +# +function fetch_kernel_from_artifactory() { + local kernel_version="$1" + local artifactory_deb="$2" + + logmust fetch_kernel_from_apt_for_version "$kernel_version" + + local url="http://artifactory.delphix.com/artifactory" + url="$url/linux-pkg/linux-prebuilt/${artifactory_deb}" + + logmust cd "$WORKDIR/artifacts" + logmust wget -nv "$url" +} + # # Determine which kernel versions to build modules for and store # the value into KERNEL_VERSIONS, unless it is already set. # -# We determine the target kernel versions based on the value passed for -# TARGET_PLATFORMS. Here is a list of accepted values for TARGET_PLATFORMS: -# a) : to build for all supported platforms -# b) "aws gcp ...": to build for the default kernel version of those platforms. -# c) "4.15.0-1010-aws ...": to build for specific kernel versions -# d) mix of b) and c) +# We determine the target kernel versions based on the kernel package +# dependencies fetched through fetch_dependencies(). # function determine_target_kernels() { if [[ -n "$KERNEL_VERSIONS" ]]; then - echo "Kernel versions to use to build modules:" - echo " $KERNEL_VERSIONS" + echo_bold "Kernel versions to use to build modules:" + echo_bold " $KERNEL_VERSIONS" return 0 fi - local supported_platforms="generic aws gcp azure oracle" - local platform + [[ -n "$DEPDIR" ]] || die "determine_target_kernels() can only be" \ + "called after fetch_dependencies() stage has run." - if [[ -z "$TARGET_PLATFORMS" ]]; then - echo "TARGET_PLATFORMS not set, defaulting to: $supported_platforms" - TARGET_PLATFORMS="$supported_platforms" - fi + logmust list_linux_kernel_packages + # note: list of kernel packages returned in _RET_LIST - local kernel - for kernel in $TARGET_PLATFORMS; do - for platform in $supported_platforms; do - if [[ "$kernel" == "$platform" ]]; then - logmust get_kernel_for_platform "$platform" - kernel="$_RET" - break - fi - done + local pkg kernel + for pkg in "${_RET_LIST[@]}"; do + logmust test -d "$DEPDIR/$pkg" # - # Check that the target kernel is valid + # When Linux kernel packages are built, they must store the + # kernel version into a file named 'KERNEL_VERSION'. # - apt-cache show "linux-image-${kernel}" >/dev/null 2>&1 || - die "Invalid target kernel '$kernel'" - + (logmust test -f "$DEPDIR/$pkg/KERNEL_VERSION") || + die "KERNEL_VERSION file missing from dependency '$pkg'" + kernel="$(cat "$DEPDIR/$pkg/KERNEL_VERSION")" + [[ -n "$kernel" ]] || die "invalid value '$kernel'" \ + "in $DEPDIR/$pkg/KERNEL_VERSION" KERNEL_VERSIONS="$KERNEL_VERSIONS $kernel" + KERNEL_VERSIONS_METADATA="${KERNEL_VERSIONS_METADATA}${pkg}: ${kernel}\\n" done - echo "Kernel versions to use to build modules:" - echo " $KERNEL_VERSIONS" + echo_bold "Kernel versions to use to build modules:" + echo_bold " $KERNEL_VERSIONS" } # @@ -854,3 +1094,51 @@ function install_gcc8() { logmust sudo update-alternatives --install /usr/bin/gcc gcc \ /usr/bin/gcc-8 800 --slave /usr/bin/g++ g++ /usr/bin/g++-8 } + +# +# Store git-related build info for the package after the build is done. +# Note that some of this metadata is used by the Jenkins build so be careful +# when modifying it. +# +function store_git_info() { + logmust pushd "$WORKDIR/repo" + local git_hash + git_hash="$(git rev-parse HEAD)" || die "Failed retrieving git hash" + echo "$git_hash" >"$WORKDIR/artifacts/GIT_HASH" + + cat <<-EOF >>"$WORKDIR/artifacts/BUILD_INFO" + Git hash: $git_hash + Git repo: $PACKAGE_GIT_URL + Git branch: $PACKAGE_GIT_BRANCH + EOF + logmust popd +} + +# +# Store build info metadata for the package after the build is done. +# Note that some of this metadata is used by the Jenkins build so be careful +# when modifying it. +# +function store_build_info() { + if [[ -d "$WORKDIR/repo/.git" ]]; then + logmust store_git_info + fi + + if [[ -n "$KERNEL_VERSIONS_METADATA" ]]; then + echo -ne "$KERNEL_VERSIONS_METADATA" >"$WORKDIR/artifacts/KERNEL_VERSIONS" || + die 'Failed to store kernel versions metadata' + fi + + if [[ -n "$PACKAGE_DEPENDENCIES_METADATA" ]]; then + echo -ne "$PACKAGE_DEPENDENCIES_METADATA" >"$WORKDIR/artifacts/PACKAGE_DEPENDENCIES" || + die 'Failed to store package dependencies metadata' + fi + + if [[ -f "$TOP/PACKAGE_MIRROR_URL_MAIN" ]]; then + logmust cp "$TOP/PACKAGE_MIRROR_URL_MAIN" "$WORKDIR/artifacts/" + fi + + if [[ -f "$TOP/PACKAGE_MIRROR_URL_SECONDARY" ]]; then + logmust cp "$TOP/PACKAGE_MIRROR_URL_SECONDARY" "$WORKDIR/artifacts/" + fi +} diff --git a/package-lists/auto-merge-blacklist.pkgs b/package-lists/auto-merge-blacklist.pkgs deleted file mode 100644 index 1268325e..00000000 --- a/package-lists/auto-merge-blacklist.pkgs +++ /dev/null @@ -1,5 +0,0 @@ -# -# Packages in this list will not be auto-merged with upstream, even if they -# are listed in updatelist.pkgs. A package should be added here only if the -# latest version of an upstream package breaks the delphix appliance. -# diff --git a/package-lists/build/adoptopenjdk.pkgs b/package-lists/build/adoptopenjdk.pkgs deleted file mode 100644 index 36c66eec..00000000 --- a/package-lists/build/adoptopenjdk.pkgs +++ /dev/null @@ -1,7 +0,0 @@ -# -# This list is responsible of building adoptopenjdk. We first need to build -# our modified make-jpkg package. -# - -make-jpkg -adoptopenjdk diff --git a/package-lists/build/kernel.pkgs b/package-lists/build/kernel.pkgs deleted file mode 100644 index 50be040f..00000000 --- a/package-lists/build/kernel.pkgs +++ /dev/null @@ -1,14 +0,0 @@ -# -# This build list includes all the packages that are build for specific -# kernel versions. -# - -# Note: The following packages should be built first because other packages -# depend on them being built. -# - zfs is required by grub2 and recovery-environment -zfs - -connstat -delphix-kernel -grub2 -recovery-environment diff --git a/package-lists/build/td-agent.pkgs b/package-lists/build/td-agent.pkgs deleted file mode 100644 index 92b741bb..00000000 --- a/package-lists/build/td-agent.pkgs +++ /dev/null @@ -1,8 +0,0 @@ -# -# Since building td-agent is currently quite unreliable as it depends on -# lots of external dependencies that are out of our control, we do not want -# to fail our build when some of those dependencies are broken. As such, -# we build td-agent in a separate build list and include a pre-built package -# in the main list. -# -td-agent diff --git a/package-lists/build/userland.pkgs b/package-lists/build/userland.pkgs deleted file mode 100644 index 0220d2f4..00000000 --- a/package-lists/build/userland.pkgs +++ /dev/null @@ -1,33 +0,0 @@ -# -# List of misc userland packages to be included in the Delphix Appliance. -# - -# Note: The following packages should be built first because other packages -# depend on them being built. -# - bcc is required by bpftrace -# - java8 is required by the saml app -bcc -java8 -make-jpkg -adoptopenjdk -libkdumpfile - -delphix-sso-app -bpftrace -challenge-response -cloud-init -crash -crash-python -crypt-blowfish -delphix-platform -drgn -gdb-python -makedumpfile -nfs-utils -python-rtslib-fb -savedump -sdb -targetcli-fb -performance-diagnostics -ptools -td-agent-prebuilt diff --git a/package-lists/update/userland.pkgs b/package-lists/update/userland.pkgs deleted file mode 100644 index c9b588cb..00000000 --- a/package-lists/update/userland.pkgs +++ /dev/null @@ -1,23 +0,0 @@ -# -# All those packages will be auto-updated by the linux-pkg-update/userland job. -# Note that if you want to keep a package's upstream branch to be updated -# but prevent it from being merged, add the package to -# auto-merge-blacklist.pkg. -# - -# Note: we do not auto-update bcc for now as upstream tends to introduce -# changes that break bpftrace. Instead, we manually update to tagged versions. -# In the future we may want to automatically determine the latest tag and -# auto-update to that. -#bcc - -bpftrace -cloud-init -crash -drgn -libkdumpfile -makedumpfile -nfs-utils -python-rtslib-fb -targetcli-fb -grub2 diff --git a/package-lists/update/zfs.pkgs b/package-lists/update/zfs.pkgs deleted file mode 100644 index e5c56b71..00000000 --- a/package-lists/update/zfs.pkgs +++ /dev/null @@ -1,6 +0,0 @@ -# -# ZFS is configured to be updated every time we detect an upstream change, -# so we provide a separate update list for it. -# - -zfs diff --git a/packages/adoptopenjdk/config.sh b/packages/adoptopenjdk/config.sh index 84eab59d..eb14a272 100755 --- a/packages/adoptopenjdk/config.sh +++ b/packages/adoptopenjdk/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -23,10 +23,7 @@ tarfile="OpenJDK8U-jdk_x64_linux_hotspot_8u262b10.tar.gz" jdk_path="/usr/lib/jvm/adoptopenjdk-java8-jdk-amd64" function prepare() { - if ! ls "$TOP/packages/make-jpkg/tmp/artifacts/"*deb >/dev/null 2>&1; then - echo_bold "custom java-package not installed. Building package 'make-jpkg' first." - logmust "$TOP/buildpkg.sh" make-jpkg - fi + logmust install_pkgs "$DEPDIR"/make-jpkg/*.deb } function fetch() { @@ -53,13 +50,6 @@ function build() { # the Linux-pkg bundle. # logmust bash -c "echo $jdk_path >'$WORKDIR/artifacts/JDK_PATH'" - # - # Install the Java package on this system so that other linux-pkg - # packages can use it. - # - logmust install_pkgs "$WORKDIR/artifacts/"*.deb -} -function store_build_info() { - echo "Tar file: $tarfile" >"$WORKDIR/build_info" + echo "Tar file: $tarfile" >"$WORKDIR/artifacts/BUILD_INFO" } diff --git a/packages/bcc/config.sh b/packages/bcc/config.sh index 26b4bf0f..c7717bcf 100644 --- a/packages/bcc/config.sh +++ b/packages/bcc/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -32,9 +32,6 @@ function build() { PACKAGE_VERSION=$(dpkg-parsechangelog | sed -rne 's,^Version: (.*),\1,p') logmust dpkg_buildpackage_default - - # Install libbcc which is required to build bpftrace - logmust install_pkgs "$WORKDIR/artifacts"/libbcc_*.deb } function update_upstream() { diff --git a/packages/bpftrace/config.sh b/packages/bpftrace/config.sh index ef2f6360..52c53459 100644 --- a/packages/bpftrace/config.sh +++ b/packages/bpftrace/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -24,11 +24,7 @@ UPSTREAM_GIT_URL="https://github.com/iovisor/bpftrace.git" UPSTREAM_GIT_BRANCH="master" function prepare() { - if ! dpkg-query --show libbcc >/dev/null 2>&1; then - echo_bold "libbcc not installed. Building package 'bcc' first." - logmust "$TOP/buildpkg.sh" bcc - fi - + logmust install_pkgs "$DEPDIR"/bcc/libbcc_*.deb logmust install_build_deps_from_control_file } diff --git a/packages/cloud-init/config.sh b/packages/cloud-init/config.sh index c40fb0ba..35b8b774 100644 --- a/packages/cloud-init/config.sh +++ b/packages/cloud-init/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2019 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/packages/connstat/config.sh b/packages/connstat/config.sh index 8bb5e3a9..cc53ed29 100644 --- a/packages/connstat/config.sh +++ b/packages/connstat/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/packages/crash-python/config.sh b/packages/crash-python/config.sh index 9449bb2e..f3a58eec 100644 --- a/packages/crash-python/config.sh +++ b/packages/crash-python/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2019 Delphix +# Copyright 2019, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/packages/crypt-blowfish/config.sh b/packages/crypt-blowfish/config.sh index 1f020ad5..cc3e3a97 100644 --- a/packages/crypt-blowfish/config.sh +++ b/packages/crypt-blowfish/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2019 Delphix +# Copyright 2019, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/packages/delphix-kernel/config.sh b/packages/delphix-kernel/config.sh index f464dfda..e7a7848c 100644 --- a/packages/delphix-kernel/config.sh +++ b/packages/delphix-kernel/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/packages/delphix-platform/config.sh b/packages/delphix-platform/config.sh index 69a6cc08..d070ffc3 100644 --- a/packages/delphix-platform/config.sh +++ b/packages/delphix-platform/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/packages/delphix-sso-app/config.sh b/packages/delphix-sso-app/config.sh index 956fdb6d..d3fcd2b2 100644 --- a/packages/delphix-sso-app/config.sh +++ b/packages/delphix-sso-app/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2019 Delphix +# Copyright 2019, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -17,20 +17,16 @@ # shellcheck disable=SC2034 DEFAULT_PACKAGE_GIT_URL="https://gitlab.delphix.com/app/saml-app.git" -JDK_PATH_FILE="$TOP/packages/adoptopenjdk/tmp/artifacts/JDK_PATH" PACKAGE_DEPENDENCIES="adoptopenjdk" function prepare() { - java_package_exists=$(dpkg-query --show adoptopenjdk-java8-jdk >/dev/null 2>&1) - if [[ ! $java_package_exists && ! -f $JDK_PATH_FILE ]]; then - echo_bold "java8 not installed. Building package 'adoptopenjdk' first." - logmust "$TOP/buildpkg.sh" adoptopenjdk - fi + logmust install_pkgs "$DEPDIR"/adoptopenjdk/*.deb } function build() { local java_home - java_home=$(cat "$JDK_PATH_FILE") + java_home=$(cat "$DEPDIR/adoptopenjdk/JDK_PATH") || + die "Failed to read $DEPDIR/adoptopenjdk/JDK_PATH" logmust cd "$WORKDIR/repo" logmust sudo ./gradlew "-Dorg.gradle.java.home=$java_home" distDeb logmust sudo mv ./build/distributions/*deb "$WORKDIR/artifacts/" diff --git a/packages/drgn/config.sh b/packages/drgn/config.sh index 737263ac..d0d1adab 100644 --- a/packages/drgn/config.sh +++ b/packages/drgn/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2019 Delphix +# Copyright 2019, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -29,11 +29,7 @@ function prepare() { # drgn itself, but it is a hard requirement in our use-case as # we do want to use drgn for kdump-compressed crash dumps. # - if ! dpkg-query --show libkdumpfile >/dev/null 2>&1; then - echo_bold "libkdumpfile not installed. Building package 'libkdumpfile' first." - logmust "$TOP/buildpkg.sh" libkdumpfile - fi - + logmust install_pkgs "$DEPDIR"/libkdumpfile/*.deb logmust install_build_deps_from_control_file } diff --git a/packages/gdb-python/config.sh b/packages/gdb-python/config.sh index aea4a7c2..fbe6ee93 100644 --- a/packages/gdb-python/config.sh +++ b/packages/gdb-python/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2019 Delphix +# Copyright 2019, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/packages/grub2/config.sh b/packages/grub2/config.sh index 254fe769..8807d7ff 100644 --- a/packages/grub2/config.sh +++ b/packages/grub2/config.sh @@ -28,13 +28,9 @@ SKIP_COPYRIGHTS_CHECK=true # Install build dependencies for the package. # function prepare() { - if ! dpkg-query --show libzfslinux-dev >/dev/null 2>&1; then - echo_bold "libzfs not installed. Building package 'zfs' first." - logmust "$TOP/buildpkg.sh" zfs - fi - + # Install libzfs which is required to build grub + logmust install_pkgs "$DEPDIR"/zfs/{libnvpair1linux,libuutil1linux,libzfs2linux,libzpool2linux,libzfslinux-dev}_*.deb logmust install_build_deps_from_control_file - return } # @@ -54,5 +50,4 @@ function build() { # function update_upstream() { logmust update_upstream_from_git - return } diff --git a/packages/java8/config.sh b/packages/java8/config.sh index 39f98d39..5d03baeb 100644 --- a/packages/java8/config.sh +++ b/packages/java8/config.sh @@ -45,13 +45,6 @@ function build() { # the Linux-pkg bundle. # logmust bash -c "echo $jdk_path >'$WORKDIR/artifacts/JDK_PATH'" - # - # Install the Java package on this system so that other linux-pkg - # packages can use it. - # - logmust install_pkgs "$WORKDIR/artifacts/"*.deb -} -function store_build_info() { - echo "Tar file: $tarfile" >"$WORKDIR/build_info" + echo "Tar file: $tarfile" >"$WORKDIR/artifacts/BUILD_INFO" } diff --git a/packages/libkdumpfile/config.sh b/packages/libkdumpfile/config.sh index 7710e2d8..a42c82ef 100644 --- a/packages/libkdumpfile/config.sh +++ b/packages/libkdumpfile/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2019 Delphix +# Copyright 2019, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -28,9 +28,6 @@ function prepare() { function build() { logmust dpkg_buildpackage_default - - # Install libkdumpfile, it's needed to build drgn - logmust install_pkgs "$WORKDIR/artifacts"/*.deb } function update_upstream() { diff --git a/build-info-pkg/debian/rules b/packages/linux-kernel-aws/config.archive.sh old mode 100755 new mode 100644 similarity index 70% rename from build-info-pkg/debian/rules rename to packages/linux-kernel-aws/config.archive.sh index a4c3e1d1..c8b581d8 --- a/build-info-pkg/debian/rules +++ b/packages/linux-kernel-aws/config.archive.sh @@ -1,6 +1,6 @@ -#!/usr/bin/make -f +#!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,14 +14,15 @@ # See the License for the specific language governing permissions and # limitations under the License. # +# shellcheck disable=SC2034 -%: - dh $@ +DEFAULT_PACKAGE_GIT_URL="none" -override_dh_auto_test: - # - # Don't run 'make test' during the build step; we'll enforce - # testing via another mechanism, so running it during package - # builds is unnecessary. - # +function fetch() { + # Nothing to do + return +} +function build() { + logmust fetch_kernel_from_apt_for_platform aws +} diff --git a/packages/linux-kernel-aws/config.delphix.sh b/packages/linux-kernel-aws/config.delphix.sh new file mode 100644 index 00000000..5d7a2ccc --- /dev/null +++ b/packages/linux-kernel-aws/config.delphix.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# shellcheck disable=SC2034 +DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/linux-kernel-aws.git" + +UPSTREAM_GIT_URL="https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-aws/+git/bionic" +# Note: UPSTREAM_GIT_BRANCH is not used here +UPSTREAM_GIT_BRANCH="none" + +# +# Force push required when syncing with upstream because we perform a rebase. +# +FORCE_PUSH_ON_UPDATE=true + +function prepare() { + logmust kernel_prepare +} + +function build() { + logmust kernel_build "aws" +} + +function update_upstream() { + logmust kernel_update_upstream "aws" +} + +function merge_with_upstream() { + logmust kernel_merge_with_upstream +} diff --git a/build-info-pkg/debian/control.in b/packages/linux-kernel-aws/config.prebuilt.sh similarity index 64% rename from build-info-pkg/debian/control.in rename to packages/linux-kernel-aws/config.prebuilt.sh index a8f07dc8..55067bee 100644 --- a/build-info-pkg/debian/control.in +++ b/packages/linux-kernel-aws/config.prebuilt.sh @@ -1,5 +1,6 @@ +#!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,14 +14,16 @@ # See the License for the specific language governing permissions and # limitations under the License. # +# shellcheck disable=SC2034 -Source: @@PACKAGE@@ -Section: metapackages -Priority: optional -Maintainer: Delphix Engineering -Build-Depends: debhelper (>= 10) -Standards-Version: 4.1.2 +DEFAULT_PACKAGE_GIT_URL="none" -Package: @@PACKAGE@@ -Architecture: any -Description: Provides build metadata for packages built by linux-pkg. +function fetch() { + # Nothing to do + return +} + +function build() { + logmust fetch_kernel_from_artifactory "5.3.0-1033-aws" \ + "6.0.5.0/dx1/linux-modules-5.3.0-1033-aws_5.3.0-1033.dx1_amd64.deb" +} diff --git a/packages/linux-kernel-aws/config.sh b/packages/linux-kernel-aws/config.sh index b42417bd..3a8c5eaa 100644 --- a/packages/linux-kernel-aws/config.sh +++ b/packages/linux-kernel-aws/config.sh @@ -15,20 +15,19 @@ # limitations under the License. # -# shellcheck disable=SC2034 -DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/linux-kernel-aws.git" - -UPSTREAM_GIT_URL="https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-aws/+git/bionic" -UPSTREAM_GIT_BRANCH="@PLACEHOLDER-WORKAROUND@" - -function prepare() { - kernel_prepare -} - -function build() { - kernel_build "aws" -} +# +# We currently support getting the linux kernel from 3 different sources: +# 1. Building it from code: see config.delphix.sh +# 2. Dowloading from apt: see config.archive.sh +# 3. Pre-built kernel stored in artifactory: see config.prebuilt.sh +# -function update_upstream() { - kernel_update_upstream "aws" -} +linux_package_source="${LINUX_KERNEL_PACKAGE_SOURCE:-$DEFAULT_LINUX_KERNEL_PACKAGE_SOURCE}" +case "$linux_package_source" in +delphix | archive | prebuilt) + logmust source "${BASH_SOURCE%/*}/config.${linux_package_source}.sh" + ;; +default) + die "invalid linux-kernel package source '$linux_package_source'" + ;; +esac diff --git a/build-info-pkg/clean.sh b/packages/linux-kernel-azure/config.archive.sh old mode 100755 new mode 100644 similarity index 67% rename from build-info-pkg/clean.sh rename to packages/linux-kernel-azure/config.archive.sh index 7108de5d..e84cb7ee --- a/build-info-pkg/clean.sh +++ b/packages/linux-kernel-azure/config.archive.sh @@ -1,6 +1,6 @@ -#!/bin/bash -eu +#!/bin/bash # -# Copyright 2019 Delphix +# Copyright 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,14 +14,15 @@ # See the License for the specific language governing permissions and # limitations under the License. # +# shellcheck disable=SC2034 -cd "$(dirname "${BASH_SOURCE[0]}")" +DEFAULT_PACKAGE_GIT_URL="none" -rm -rf lib -rm -rf artifacts -rm -f debian/control -rm -f debian/changelog -rm -rf debian/.debhelper/ -rm -rf debian/delphix-buildinfo* -rm -f debian/files -rm -f debian/debhelper-build-stamp +function fetch() { + # Nothing to do + return +} + +function build() { + logmust fetch_kernel_from_apt_for_platform "azure" +} diff --git a/packages/linux-kernel-azure/config.delphix.sh b/packages/linux-kernel-azure/config.delphix.sh new file mode 100644 index 00000000..a6bcc23a --- /dev/null +++ b/packages/linux-kernel-azure/config.delphix.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# shellcheck disable=SC2034 +DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/linux-kernel-azure.git" + +UPSTREAM_GIT_URL="https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-azure/+git/bionic" +# Note: UPSTREAM_GIT_BRANCH is not used here +UPSTREAM_GIT_BRANCH="none" + +# +# Force push required when syncing with upstream because we perform a rebase. +# +FORCE_PUSH_ON_UPDATE=true + +function prepare() { + logmust kernel_prepare +} + +function build() { + logmust kernel_build "azure" +} + +function update_upstream() { + logmust kernel_update_upstream "azure" +} + +function merge_with_upstream() { + logmust kernel_merge_with_upstream +} diff --git a/packages/linux-kernel-azure/config.prebuilt.sh b/packages/linux-kernel-azure/config.prebuilt.sh new file mode 100644 index 00000000..5c02477f --- /dev/null +++ b/packages/linux-kernel-azure/config.prebuilt.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# shellcheck disable=SC2034 + +DEFAULT_PACKAGE_GIT_URL="none" + +function fetch() { + # Nothing to do + return +} + +function build() { + logmust fetch_kernel_from_artifactory "5.3.0-1035-azure" \ + "6.0.5.0/dx1/linux-modules-5.3.0-1035-azure_5.3.0-1035.dx1_amd64.deb" +} diff --git a/packages/linux-kernel-azure/config.sh b/packages/linux-kernel-azure/config.sh index ddb8c34d..3a8c5eaa 100644 --- a/packages/linux-kernel-azure/config.sh +++ b/packages/linux-kernel-azure/config.sh @@ -15,20 +15,19 @@ # limitations under the License. # -# shellcheck disable=SC2034 -DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/linux-kernel-azure.git" - -UPSTREAM_GIT_URL="https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-azure/+git/bionic" -UPSTREAM_GIT_BRANCH="@PLACEHOLDER-WORKAROUND@" - -function prepare() { - kernel_prepare -} - -function build() { - kernel_build "azure" -} +# +# We currently support getting the linux kernel from 3 different sources: +# 1. Building it from code: see config.delphix.sh +# 2. Dowloading from apt: see config.archive.sh +# 3. Pre-built kernel stored in artifactory: see config.prebuilt.sh +# -function update_upstream() { - kernel_update_upstream "azure" -} +linux_package_source="${LINUX_KERNEL_PACKAGE_SOURCE:-$DEFAULT_LINUX_KERNEL_PACKAGE_SOURCE}" +case "$linux_package_source" in +delphix | archive | prebuilt) + logmust source "${BASH_SOURCE%/*}/config.${linux_package_source}.sh" + ;; +default) + die "invalid linux-kernel package source '$linux_package_source'" + ;; +esac diff --git a/packages/linux-kernel-gcp/config.archive.sh b/packages/linux-kernel-gcp/config.archive.sh new file mode 100644 index 00000000..f39a4006 --- /dev/null +++ b/packages/linux-kernel-gcp/config.archive.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# shellcheck disable=SC2034 + +DEFAULT_PACKAGE_GIT_URL="none" + +function fetch() { + # Nothing to do + return +} + +function build() { + logmust fetch_kernel_from_apt_for_platform "gcp" +} diff --git a/packages/linux-kernel-gcp/config.delphix.sh b/packages/linux-kernel-gcp/config.delphix.sh new file mode 100644 index 00000000..342547c3 --- /dev/null +++ b/packages/linux-kernel-gcp/config.delphix.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# shellcheck disable=SC2034 +DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/linux-kernel-gcp.git" + +UPSTREAM_GIT_URL="https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-gcp/+git/bionic" +# Note: UPSTREAM_GIT_BRANCH is not used here +UPSTREAM_GIT_BRANCH="none" + +# +# Force push required when syncing with upstream because we perform a rebase. +# +FORCE_PUSH_ON_UPDATE=true + +function prepare() { + logmust kernel_prepare +} + +function build() { + logmust kernel_build "gcp" +} + +function update_upstream() { + logmust kernel_update_upstream "gcp" +} + +function merge_with_upstream() { + logmust kernel_merge_with_upstream +} diff --git a/packages/linux-kernel-gcp/config.prebuilt.sh b/packages/linux-kernel-gcp/config.prebuilt.sh new file mode 100644 index 00000000..0ac0f4b1 --- /dev/null +++ b/packages/linux-kernel-gcp/config.prebuilt.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# shellcheck disable=SC2034 + +DEFAULT_PACKAGE_GIT_URL="none" + +function fetch() { + # Nothing to do + return +} + +function build() { + logmust fetch_kernel_from_artifactory "5.4.0-1021-gcp" \ + "6.0.5.0/dx1/linux-modules-5.4.0-1021-gcp_5.4.0-1021.dx1_amd64.deb" +} diff --git a/packages/linux-kernel-gcp/config.sh b/packages/linux-kernel-gcp/config.sh index 2ba303b7..3a8c5eaa 100644 --- a/packages/linux-kernel-gcp/config.sh +++ b/packages/linux-kernel-gcp/config.sh @@ -15,20 +15,19 @@ # limitations under the License. # -# shellcheck disable=SC2034 -DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/linux-kernel-gcp.git" - -UPSTREAM_GIT_URL="https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-gcp/+git/bionic" -UPSTREAM_GIT_BRANCH="@PLACEHOLDER-WORKAROUND@" - -function prepare() { - kernel_prepare -} - -function build() { - kernel_build "gcp" -} +# +# We currently support getting the linux kernel from 3 different sources: +# 1. Building it from code: see config.delphix.sh +# 2. Dowloading from apt: see config.archive.sh +# 3. Pre-built kernel stored in artifactory: see config.prebuilt.sh +# -function update_upstream() { - kernel_update_upstream "gcp" -} +linux_package_source="${LINUX_KERNEL_PACKAGE_SOURCE:-$DEFAULT_LINUX_KERNEL_PACKAGE_SOURCE}" +case "$linux_package_source" in +delphix | archive | prebuilt) + logmust source "${BASH_SOURCE%/*}/config.${linux_package_source}.sh" + ;; +default) + die "invalid linux-kernel package source '$linux_package_source'" + ;; +esac diff --git a/packages/linux-kernel-generic/config.archive.sh b/packages/linux-kernel-generic/config.archive.sh new file mode 100644 index 00000000..124f19f7 --- /dev/null +++ b/packages/linux-kernel-generic/config.archive.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# shellcheck disable=SC2034 + +DEFAULT_PACKAGE_GIT_URL="none" + +function fetch() { + # Nothing to do + return +} + +function build() { + logmust fetch_kernel_from_apt_for_platform "generic" +} diff --git a/packages/linux-kernel-generic/config.delphix.sh b/packages/linux-kernel-generic/config.delphix.sh new file mode 100644 index 00000000..4db55af2 --- /dev/null +++ b/packages/linux-kernel-generic/config.delphix.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# shellcheck disable=SC2034 +DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/linux-kernel-generic.git" + +UPSTREAM_GIT_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic" +# Note: UPSTREAM_GIT_BRANCH is not used here +UPSTREAM_GIT_BRANCH="none" + +# +# Force push required when syncing with upstream because we perform a rebase. +# +FORCE_PUSH_ON_UPDATE=true + +function prepare() { + logmust kernel_prepare +} + +function build() { + logmust kernel_build "generic" +} + +function update_upstream() { + logmust kernel_update_upstream "generic" +} + +function merge_with_upstream() { + logmust kernel_merge_with_upstream +} diff --git a/packages/linux-kernel-generic/config.prebuilt.sh b/packages/linux-kernel-generic/config.prebuilt.sh new file mode 100644 index 00000000..ed700f7e --- /dev/null +++ b/packages/linux-kernel-generic/config.prebuilt.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# shellcheck disable=SC2034 + +DEFAULT_PACKAGE_GIT_URL="none" + +function fetch() { + # Nothing to do + return +} + +function build() { + logmust fetch_kernel_from_artifactory "5.4.0-42-generic" \ + "6.0.5.0/dx1/linux-modules-5.4.0-42-generic_5.4.0-42.dx1_amd64.deb" +} diff --git a/packages/linux-kernel-generic/config.sh b/packages/linux-kernel-generic/config.sh index baa0708c..3a8c5eaa 100644 --- a/packages/linux-kernel-generic/config.sh +++ b/packages/linux-kernel-generic/config.sh @@ -15,26 +15,19 @@ # limitations under the License. # -# shellcheck disable=SC2034 -DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/linux-kernel-generic.git" - -UPSTREAM_GIT_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic" -UPSTREAM_GIT_BRANCH="@PLACEHOLDER-WORKAROUND@" - -function prepare() { - kernel_prepare -} - -function build() { - # - # flavours=generic - # By default the generic kernel variant from Canonical - # builds both the generic and the low-latency kernel. - # We don't care about the latter. - # - kernel_build "generic" "flavours=generic" -} +# +# We currently support getting the linux kernel from 3 different sources: +# 1. Building it from code: see config.delphix.sh +# 2. Dowloading from apt: see config.archive.sh +# 3. Pre-built kernel stored in artifactory: see config.prebuilt.sh +# -function update_upstream() { - kernel_update_upstream "generic" -} +linux_package_source="${LINUX_KERNEL_PACKAGE_SOURCE:-$DEFAULT_LINUX_KERNEL_PACKAGE_SOURCE}" +case "$linux_package_source" in +delphix | archive | prebuilt) + logmust source "${BASH_SOURCE%/*}/config.${linux_package_source}.sh" + ;; +default) + die "invalid linux-kernel package source '$linux_package_source'" + ;; +esac diff --git a/packages/linux-kernel-oracle/config.archive.sh b/packages/linux-kernel-oracle/config.archive.sh new file mode 100644 index 00000000..6c283a3f --- /dev/null +++ b/packages/linux-kernel-oracle/config.archive.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# shellcheck disable=SC2034 + +DEFAULT_PACKAGE_GIT_URL="none" + +function fetch() { + # Nothing to do + return +} + +function build() { + logmust fetch_kernel_from_apt_for_platform "oracle" +} diff --git a/packages/linux-kernel-oracle/config.delphix.sh b/packages/linux-kernel-oracle/config.delphix.sh new file mode 100644 index 00000000..0bd27633 --- /dev/null +++ b/packages/linux-kernel-oracle/config.delphix.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# shellcheck disable=SC2034 +DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/linux-kernel-oracle.git" + +UPSTREAM_GIT_URL="https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-oracle/+git/bionic" +# Note: UPSTREAM_GIT_BRANCH is not used here +UPSTREAM_GIT_BRANCH="none" + +# +# Force push required when syncing with upstream because we perform a rebase. +# +FORCE_PUSH_ON_UPDATE=true + +function prepare() { + logmust kernel_prepare +} + +function build() { + logmust kernel_build "oracle" +} + +function update_upstream() { + logmust kernel_update_upstream "oracle" +} + +function merge_with_upstream() { + logmust kernel_merge_with_upstream +} diff --git a/packages/linux-kernel-oracle/config.prebuilt.sh b/packages/linux-kernel-oracle/config.prebuilt.sh new file mode 100644 index 00000000..82115eb6 --- /dev/null +++ b/packages/linux-kernel-oracle/config.prebuilt.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# shellcheck disable=SC2034 + +DEFAULT_PACKAGE_GIT_URL="none" + +function fetch() { + # Nothing to do + return +} + +function build() { + logmust fetch_kernel_from_artifactory "5.4.0-1021-oracle" \ + "6.0.5.0/dx1/linux-modules-5.4.0-1021-oracle_5.4.0-1021.dx1_amd64.deb" +} diff --git a/packages/linux-kernel-oracle/config.sh b/packages/linux-kernel-oracle/config.sh index 85b1a102..3a8c5eaa 100644 --- a/packages/linux-kernel-oracle/config.sh +++ b/packages/linux-kernel-oracle/config.sh @@ -15,20 +15,19 @@ # limitations under the License. # -# shellcheck disable=SC2034 -DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/linux-kernel-oracle.git" - -UPSTREAM_GIT_URL="https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-oracle/+git/bionic" -UPSTREAM_GIT_BRANCH="@PLACEHOLDER-WORKAROUND@" - -function prepare() { - kernel_prepare -} - -function build() { - kernel_build "oracle" -} +# +# We currently support getting the linux kernel from 3 different sources: +# 1. Building it from code: see config.delphix.sh +# 2. Dowloading from apt: see config.archive.sh +# 3. Pre-built kernel stored in artifactory: see config.prebuilt.sh +# -function update_upstream() { - kernel_update_upstream "oracle" -} +linux_package_source="${LINUX_KERNEL_PACKAGE_SOURCE:-$DEFAULT_LINUX_KERNEL_PACKAGE_SOURCE}" +case "$linux_package_source" in +delphix | archive | prebuilt) + logmust source "${BASH_SOURCE%/*}/config.${linux_package_source}.sh" + ;; +default) + die "invalid linux-kernel package source '$linux_package_source'" + ;; +esac diff --git a/packages/make-jpkg/config.sh b/packages/make-jpkg/config.sh index 6aab3f8f..42c1cb5d 100644 --- a/packages/make-jpkg/config.sh +++ b/packages/make-jpkg/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2019 Delphix +# Copyright 2019, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -32,11 +32,6 @@ function build() { fi logmust dpkg_buildpackage_default - - # java-package supporting adoptopenjdk needs to be installed to - # create jdk debian package from jdk tarball when building adoptopenjdk - # package - logmust install_pkgs "$WORKDIR/artifacts"/*.deb } function update_upstream() { diff --git a/packages/makedumpfile/config.sh b/packages/makedumpfile/config.sh index 03676b38..398105b4 100644 --- a/packages/makedumpfile/config.sh +++ b/packages/makedumpfile/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2019 Delphix +# Copyright 2019, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/packages/ptools/config.sh b/packages/ptools/config.sh index 2c637002..ca7edbc6 100644 --- a/packages/ptools/config.sh +++ b/packages/ptools/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2019 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/packages/recovery-environment/config.sh b/packages/recovery-environment/config.sh index 1ca33944..74e1c049 100644 --- a/packages/recovery-environment/config.sh +++ b/packages/recovery-environment/config.sh @@ -20,16 +20,10 @@ DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/recovery-environment.git" DEFAULT_PACKAGE_VERSION=1.0.0 PACKAGE_DEPENDENCIES="zfs" -ZFS_DEB_PATH="$TOP/packages/zfs/tmp/artifacts" function prepare() { - if [ ! "$(ls -A "$ZFS_DEB_PATH")" ]; then - logmust "$TOP/buildpkg.sh" zfs - fi - logmust install_build_deps_from_control_file logmust mkdir "$WORKDIR/repo/external-debs/" - logmust cp "$ZFS_DEB_PATH/"*.deb "$WORKDIR/repo/external-debs/" - return + logmust cp "$DEPDIR/zfs/"*.deb "$WORKDIR/repo/external-debs/" } function build() { diff --git a/packages/targetcli-fb/config.sh b/packages/targetcli-fb/config.sh index 7b6f406a..4d06dd39 100644 --- a/packages/targetcli-fb/config.sh +++ b/packages/targetcli-fb/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -23,6 +23,7 @@ PACKAGE_DEPENDENCIES="python-rtslib-fb" UPSTREAM_SOURCE_PACKAGE=targetcli-fb function prepare() { + logmust install_pkgs "$DEPDIR"/python-rtslib-fb/python3-rtslib-fb*.deb logmust install_build_deps_from_control_file } diff --git a/packages/td-agent-prebuilt/config.sh b/packages/td-agent-prebuilt/config.sh index 4fef2cc7..8e5e2156 100644 --- a/packages/td-agent-prebuilt/config.sh +++ b/packages/td-agent-prebuilt/config.sh @@ -18,11 +18,10 @@ DEFAULT_PACKAGE_GIT_URL=none SKIP_COPYRIGHTS_CHECK=true -package="td-agent_3.5.0-delphix-2019.09.18.20_amd64.deb" function fetch() { logmust cd "$WORKDIR/artifacts" - + local package="td-agent_3.5.0-delphix-2019.09.18.20_amd64.deb" local url="http://artifactory.delphix.com/artifactory" logmust wget -nv "$url/linux-pkg/td-agent/$package" -O "$package" diff --git a/packages/zfs/config.sh b/packages/zfs/config.sh index 6d3f6956..3140045e 100644 --- a/packages/zfs/config.sh +++ b/packages/zfs/config.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2019 Delphix +# Copyright 2019, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -163,9 +163,6 @@ function build() { done logmust cd "$WORKDIR" logmust mv "all-packages/"*.deb "artifacts/" - - # Install libzfs which is required to build grub - logmust install_pkgs "$WORKDIR/artifacts"/{libnvpair1linux,libuutil1linux,libzfs2linux,libzpool2linux,libzfslinux-dev}_*.deb } function update_upstream() { diff --git a/push-merge.sh b/push-merge.sh new file mode 100755 index 00000000..ec60e896 --- /dev/null +++ b/push-merge.sh @@ -0,0 +1,96 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" +source "$TOP/lib/common.sh" + +logmust check_running_system + +function usage() { + [[ $# != 0 ]] && echo "$(basename "$0"): $*" + echo "Usage: $(basename "$0") " + echo "" + echo " Push code that was previously merged. sync-with-upstream.sh must" + echo " already have been run. Before pushing the merge, it will first" + echo " check that the target branch has not been modified since the merge" + echo " was performed, and fail if it did." + echo "" + echo " As a safety check, DRYRUN environment variable must be set to" + echo " 'false'." + echo "" + echo " -h display this message and exit." + echo "" + exit 2 +} + +while getopts ':h' c; do + case "$c" in + h) usage >&2 ;; + *) usage "illegal option -- $OPTARG" >&2 ;; + esac +done +shift $((OPTIND - 1)) +[[ $# -lt 1 ]] && usage "package argument missing" >&2 +[[ $# -gt 1 ]] && usage "too many arguments" >&2 +PACKAGE=$1 + +if [[ "$DRYRUN" != 'false' ]]; then + die "DRYRUN environment variable must be set to 'false'." +fi + +logmust check_package_exists "$PACKAGE" + +DEFAULT_REVISION="${DEFAULT_REVISION:-$(default_revision)}" +logmust determine_default_git_branch +logmust load_package_config "$PACKAGE" + +if [[ ! -d "$WORKDIR/repo" ]]; then + die "$WORKDIR/repo doesn't exist, have you run sync-with-upstream for" \ + "package $PACKAGE?" +fi +logmust cd "$WORKDIR/repo" + +# +# Check that the target branch has not been modified in the meanwhile. +# This is especially important for repositories that have +# FORCE_PUSH_ON_UPDATE set to true, such as the linux kernel. The file +# WORKDIR/merge-commit-outdated will be created if that is the case to let +# the caller know this is the reason the push failed. +# +set -o pipefail +echo "Running: git rev-parse refs/heads/repo-HEAD-saved" +saved_ref=$(git rev-parse refs/heads/repo-HEAD-saved) || + die "Failed to read local ref refs/heads/repo-HEAD-saved" +echo "Running: git ls-remote $DEFAULT_PACKAGE_GIT_URL refs/heads/$DEFAULT_GIT_BRANCH" +remote_ref=$(git ls-remote "$DEFAULT_PACKAGE_GIT_URL" "refs/heads/$DEFAULT_GIT_BRANCH" | + awk '{print $1}') || + die "Failed to read remote ref refs/heads/$DEFAULT_GIT_BRANCH" +set +o pipefail + +if [[ "$saved_ref" != "$remote_ref" ]]; then + touch "$WORKDIR/merge-commit-outdated" + die "Remote branch $DEFAULT_GIT_BRANCH was modified while merge" \ + "testing was being performed. Previous hash: $saved_ref," \ + "new hash: $remote_ref. Not pushing merge." +fi + +force_push="${FORCE_PUSH_ON_UPDATE:-false}" +logmust push_to_remote "refs/heads/repo-HEAD" \ + "refs/heads/$DEFAULT_GIT_BRANCH" "$force_push" + +echo_success "Merge pushed successfully for package $PACKAGE to remote" \ + "branch $DEFAULT_GIT_BRANCH" diff --git a/push-updates.sh b/push-updates.sh deleted file mode 100755 index 0beaa31c..00000000 --- a/push-updates.sh +++ /dev/null @@ -1,139 +0,0 @@ -#!/bin/bash -# -# Copyright 2018 Delphix -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" -source "$TOP/lib/common.sh" - -logmust check_running_system - -function usage() { - [[ $# != 0 ]] && echo "$(basename "$0"): $*" - echo "Usage: $(basename "$0") -m|-u [-ny] [-g target_git_url]" - echo " [-b target_git_branch] package" - echo "" - echo " This script pushes updates generated by 'buildpkg.sh -u' to the" - echo " default remote for the package. The updates should be present" - echo " in 'packages//tmp/repo', in branch repo-HEAD for the merge" - echo " branch and upstream-HEAD for the upstream branch." - echo " Options:" - echo "" - echo " -u push updates for the upstream branch" - echo " -m push updates for the merge branch" - echo " -g repository to push to. Defaults to value of" - echo " DEFAULT_PACKAGE_GIT_URL for the package." - echo " -b branch to push to. Defaults to 'master' when passing -m" - echo " or to $REPO_UPSTREAM_BRANCH when passing -u." - echo " -n dry-run. Pass the dry-run flag to git push (git push -n)." - echo " -y do not prompt for confirmation before pushing." - echo " -h display this message and exit." - echo "" - exit 2 -} - -unset TARGET_GIT_URL -unset TARGET_GIT_BRANCH - -do_push_merge=false -do_push_upstream=false -do_ask=true -dry_run=false -while getopts ':b:g:hmnuy' c; do - case "$c" in - b) TARGET_GIT_BRANCH="$OPTARG" ;; - g) TARGET_GIT_URL="$OPTARG" ;; - m) do_push_merge=true ;; - u) do_push_upstream=true ;; - n) dry_run=true ;; - y) do_ask=false ;; - h) usage >&2 ;; - *) usage "illegal option -- $OPTARG" >&2 ;; - esac -done -shift $((OPTIND - 1)) -[[ $# -lt 1 ]] && usage "package argument missing" >&2 -[[ $# -gt 1 ]] && usage "too many arguments" >&2 -PACKAGE=$1 - -if $do_push_merge && $do_push_upstream; then - usage "-m and -u are exclusive." >&2 -elif ! ($do_push_merge || $do_push_upstream); then - usage "must pass either -m or -u." >&2 -fi - -query_git_credentials - -logmust check_package_exists "$PACKAGE" -PKGDIR="$TOP/packages/$PACKAGE" -WORKDIR="$PKGDIR/tmp" - -if [[ -z "$TARGET_GIT_URL" ]]; then - # - # load the package's config just so that we can retrieve - # DEFAULT_PACKAGE_GIT_URL. - # - logmust load_package_config "$PACKAGE" - check_env DEFAULT_PACKAGE_GIT_URL - TARGET_GIT_URL="$DEFAULT_PACKAGE_GIT_URL" - echo "TARGET_GIT_URL set to DEFAULT_PACKAGE_GIT_URL" -fi - -if [[ -z "$TARGET_GIT_BRANCH" ]]; then - if $do_push_merge; then - TARGET_GIT_BRANCH="master" - echo "TARGET_GIT_BRANCH defaults to 'master' for merge pushes." - else - TARGET_GIT_BRANCH="$REPO_UPSTREAM_BRANCH" - echo "TARGET_GIT_BRANCH defaults to '$REPO_UPSTREAM_BRANCH' for" \ - "upstream pushes." - fi -fi - -[[ "$TARGET_GIT_URL" == https://* ]] || - die "Target git url must begin with https://" - -git_url_with_creds="${TARGET_GIT_URL/https:\/\//https:\/\/${PUSH_GIT_USER}:${PUSH_GIT_PASSWORD}@}" -git_url_with_fake_creds="${TARGET_GIT_URL/https:\/\//https:\/\/:@}" - -if $do_push_merge; then - ref="refs/heads/repo-HEAD" - check_updated="$WORKDIR/repo-updated" -else - ref="refs/heads/upstream-HEAD" - check_updated="$WORKDIR/upstream-updated" -fi - -[[ -f "$check_updated" ]] || - die "$check_updated is missing, aborting push." - -logmust cd "$WORKDIR/repo" -check_git_ref "$ref" - -if $do_ask; then - echo_bold "WARNING: You are about to push to branch" \ - "$TARGET_GIT_BRANCH of $TARGET_GIT_URL." - read -rp "Do you want to proceed (y/[n])? " - if [[ "$REPLY" != "y" ]]; then - echo "Push aborted by user." - exit 1 - fi -fi - -flags="" -$dry_run && flags="-n" -echo "Running: git push $flags $git_url_with_fake_creds $ref:$TARGET_GIT_BRANCH" -git push $flags "$git_url_with_creds" "$ref:$TARGET_GIT_BRANCH" || - die "Push failed." diff --git a/resources/delphix-secondary-mirror.key b/resources/delphix-secondary-mirror.key new file mode 100644 index 00000000..96d26c61 Binary files /dev/null and b/resources/delphix-secondary-mirror.key differ diff --git a/setup.sh b/setup.sh index f07223cb..2d0b2321 100755 --- a/setup.sh +++ b/setup.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2018 Delphix +# Copyright 2018, 2020 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -25,18 +25,28 @@ logmust determine_default_git_branch # mirror url is passed in, then the latest mirror snapshot is used. # configure_apt_sources() { - local package_mirror_url='' - if [[ -n "$DELPHIX_PACKAGE_MIRROR_MAIN" ]]; then - package_mirror_url="$DELPHIX_PACKAGE_MIRROR_MAIN" - else + local package_mirror_url + local primary_url="$DELPHIX_PACKAGE_MIRROR_MAIN" + local secondary_url="$DELPHIX_PACKAGE_MIRROR_SECONDARY" + + if [[ -z "$primary_url" ]] || [[ -z "$secondary_url" ]]; then local latest_url="http://linux-package-mirror.delphix.com/" latest_url+="${DEFAULT_GIT_BRANCH}/latest/" package_mirror_url=$(curl -LfSs -o /dev/null -w '%{url_effective}' \ "$latest_url" || die "Could not curl $latest_url") - - package_mirror_url+="ubuntu" + # Remove trailing slash, if present. + package_mirror_url="${package_mirror_url%/}" + [[ -z "$primary_url" ]] && primary_url="${package_mirror_url}/ubuntu" + [[ -z "$secondary_url" ]] && secondary_url="${package_mirror_url}/ppas" fi + # + # Store the package mirror in a file so that it can be added to a + # package build's metadata via store_build_info(). + # + echo "$primary_url" >"$TOP/PACKAGE_MIRROR_URL_MAIN" + echo "$secondary_url" >"$TOP/PACKAGE_MIRROR_URL_SECONDARY" + # # Remove other sources in sources.list.d if they are present. # @@ -46,18 +56,23 @@ configure_apt_sources() { ) sudo bash -c "cat <<-EOF >/etc/apt/sources.list -deb ${package_mirror_url} ${UBUNTU_DISTRIBUTION} main restricted universe multiverse -deb-src ${package_mirror_url} ${UBUNTU_DISTRIBUTION} main restricted universe multiverse + deb ${primary_url} ${UBUNTU_DISTRIBUTION} main restricted universe multiverse + deb-src ${primary_url} ${UBUNTU_DISTRIBUTION} main restricted universe multiverse + + deb ${primary_url} ${UBUNTU_DISTRIBUTION}-updates main restricted universe multiverse + deb-src ${primary_url} ${UBUNTU_DISTRIBUTION}-updates main restricted universe multiverse + + deb ${primary_url} ${UBUNTU_DISTRIBUTION}-security main restricted universe multiverse + deb-src ${primary_url} ${UBUNTU_DISTRIBUTION}-security main restricted universe multiverse -deb ${package_mirror_url} ${UBUNTU_DISTRIBUTION}-updates main restricted universe multiverse -deb-src ${package_mirror_url} ${UBUNTU_DISTRIBUTION}-updates main restricted universe multiverse + deb ${primary_url} ${UBUNTU_DISTRIBUTION}-backports main restricted universe multiverse + deb-src ${primary_url} ${UBUNTU_DISTRIBUTION}-backports main restricted universe multiverse -deb ${package_mirror_url} ${UBUNTU_DISTRIBUTION}-security main restricted universe multiverse -deb-src ${package_mirror_url} ${UBUNTU_DISTRIBUTION}-security main restricted universe multiverse + deb ${secondary_url} ${UBUNTU_DISTRIBUTION} main multiverse universe + deb ${secondary_url} ${UBUNTU_DISTRIBUTION}-updates main multiverse universe + EOF" || die "/etc/apt/sources.list could not be updated" -deb ${package_mirror_url} ${UBUNTU_DISTRIBUTION}-backports main restricted universe multiverse -deb-src ${package_mirror_url} ${UBUNTU_DISTRIBUTION}-backports main restricted universe multiverse -EOF" || die "/etc/apt/sources.list could not be updated" + logmust sudo apt-key add "$TOP/resources/delphix-secondary-mirror.key" } logmust check_running_system diff --git a/sync-with-upstream.sh b/sync-with-upstream.sh new file mode 100755 index 00000000..1f0e2069 --- /dev/null +++ b/sync-with-upstream.sh @@ -0,0 +1,108 @@ +#!/bin/bash +# +# Copyright 2020 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" +source "$TOP/lib/common.sh" + +logmust check_running_system + +function usage() { + [[ $# != 0 ]] && echo "$(basename "$0"): $*" + echo "Usage: $(basename "$0") " + echo "" + echo " Update the upstreams branch for the package and attempt to merge" + echo " active branch with the upstream. If merge succeeds, push result" + echo " to branch projects/auto-update//merging." + echo "" + echo " This script requires the DRYRUN environment variable to be set to" + echo " either 'true' or 'false', else it will refuse to push anything" + echo " upstream. If DRYRUN is set to true, the upstreams/ branch" + echo " will not be pushed and the resulting merge will be pushed to" + echo " projects/auto-update//merging-dryrun instead." + echo "" + echo " -h display this message and exit." + echo "" + exit 2 +} + +while getopts ':h' c; do + case "$c" in + h) usage >&2 ;; + *) usage "illegal option -- $OPTARG" >&2 ;; + esac +done +shift $((OPTIND - 1)) +[[ $# -lt 1 ]] && usage "package argument missing" >&2 +[[ $# -gt 1 ]] && usage "too many arguments" >&2 +PACKAGE=$1 + +logmust check_package_exists "$PACKAGE" + +logmust determine_default_git_branch + +merging_ref="refs/heads/projects/auto-update/$DEFAULT_GIT_BRANCH/merging" +if [[ "$DRYRUN" == 'true' ]]; then + merging_ref="${merging_ref}-dryrun" +elif [[ "$DRYRUN" != 'false' ]]; then + die "DRYRUN environment variable must be set to 'true' or 'false'." +fi + +logmust load_package_config "$PACKAGE" +logmust create_workdir + +# +# Set DO_UPDATE_PACKAGE to true so that the fetch stage fetches both the +# target branch as well as the upstream branch. +# +export DO_UPDATE_PACKAGE=true +logmust cd "$WORKDIR" +stage fetch + +stage update_upstream +force_push="${FORCE_PUSH_ON_UPDATE:-false}" + +if [[ -f "$WORKDIR/upstream-updated" ]]; then + if $DRYRUN; then + echo_success "Upstream updated for package $PACKAGE" \ + "but not pushed because this is a dry-run." + else + logmust push_to_remote "refs/heads/upstream-HEAD" \ + "refs/heads/upstreams/$DEFAULT_GIT_BRANCH" "$force_push" + + if [[ -f "$WORKDIR/upstream-tag" ]]; then + echo "Note: also pushing tag from upstream." + upstream_tag="$(cat "$WORKDIR/upstream-tag")" + [[ -z "$upstream_tag" ]] && + die "tag missing in $WORKDIR/upstream-tag" + logmust push_to_remote "$upstream_tag" \ + "$upstream_tag" false + fi + + echo_success "Upstream updated for package $PACKAGE." + fi +fi + +logmust cd "$WORKDIR/repo" + +stage merge_with_upstream +if [[ -f "$WORKDIR/repo-updated" ]]; then + logmust push_to_remote "refs/heads/repo-HEAD" "$merging_ref" true + echo_success "Pushed merge commit of package $PACKAGE to ref" \ + "$merging_ref of $DEFAULT_PACKAGE_GIT_URL for testing." +else + echo_bold "Package is already up-to-date." +fi diff --git a/template/config.sh b/template/config.sh index bb9a4b66..4c15fcbc 100644 --- a/template/config.sh +++ b/template/config.sh @@ -61,7 +61,7 @@ function prepare() { # #logmust install_pkgs build-dep-pkg1 build-dep-pkg2 ... #logmust install_build_deps_from_control_file - return + echo 'insert code here' } # @@ -70,7 +70,7 @@ function prepare() { # function build() { # - # Those are the default functions to build the package: + # This is the default functions to build the package: # logmust dpkg_buildpackage_default } @@ -85,5 +85,5 @@ function update_upstream() { # #logmust update_upstream_from_source_package #logmust update_upstream_from_git - return + echo 'insert code here' } diff --git a/updatelist.sh b/updatelist.sh deleted file mode 100755 index 293dcc10..00000000 --- a/updatelist.sh +++ /dev/null @@ -1,282 +0,0 @@ -#!/bin/bash -# -# Copyright 2018, 2019 Delphix -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -# -# This script first updates a list of third-party packages with upstream -# by running "buildpkg.sh -u" on each package. If updates can be merged -# cleanly and the resulting package builds, the merge is pushed to the -# package's repository (denoted by DEFAULT_PACKAGE_GIT_URL in its config.sh). -# - -TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" -source "$TOP/lib/common.sh" - -logmust check_running_system - -function exit_hook() { - echo_error "Script has exited unexpectedly." - FAILURE=true - - if [[ -z "$STATUS_DIR" ]] || [[ ! -d "$STATUS_DIR" ]]; then - echo_error "Script failed during the setup phase." - return - fi - - report_status_all -} - -function record_failure() { - FAILURE=true - if $stop_on_failure; then - echo_error "Stopping on failure." - trap - EXIT - - report_status_all - exit 1 - fi -} - -function usage() { - [[ $# != 0 ]] && echo "$(basename "$0"): $*" - echo "Usage: $(basename "$0") [-hns] " - echo "" - echo "This script attempts to update all the packages in" - echo "package-lists/update/.pkgs." - echo "" - echo " -n dry-run. Pass the dry-run flag to git push (git push -n)." - echo " -s stop on failure. By default script continues when update" - echo " for a package fails." - echo " -h display this message and exit." - echo "" - exit 2 -} - -dry_run=false -stop_on_failure=false -while getopts ':hns' c; do - case "$c" in - s) stop_on_failure=true ;; - n) dry_run=true ;; - h) usage >&2 ;; - *) usage "illegal option -- $OPTARG" >&2 ;; - esac -done -shift $((OPTIND - 1)) -[[ $# -ne 1 ]] && usage "takes exactly one argument." >&2 - -pkg_list="$1" -logmust get_package_list_file "update" "$pkg_list" -pkg_list_file="$_RET" - -trap exit_hook EXIT -FAILURE=false - -$dry_run && echo "This is a dry-run, updates will NOT be pushed to remotes." - -logmust query_git_credentials - -logmust cd "$TOP" - -logmust make clean -STATUS_DIR="$TOP/update-status" -logmust mkdir "$STATUS_DIR" - -function report_status() { - local pkg - local status - - if $FAILURE; then - status="${FMT_RED}FAILURE${FMT_NF}" - else - status="${FMT_GREEN}SUCCESS${FMT_NF}" - fi - - echo "" - echo -e "${FMT_BOLD}Status Report: $status" - - # - # Return if there is nothing to report - # - [[ -f "$STATUS_DIR/upstream-pushed" ]] || - [[ -f "$STATUS_DIR/merge-pushed" ]] || - [[ -f "$STATUS_DIR/update-failed" ]] || - [[ -f "$STATUS_DIR/unexpected-failure" ]] || - return - - echo_bold "___________________________________________" - - if $dry_run; then - echo_bold "NOTE: This is a dry-run, updates are NOT pushed." - echo "" - fi - - if [[ -f "$STATUS_DIR/upstream-pushed" ]]; then - echo -e "${FMT_GREEN}Upstream updated for following" \ - "packages:${FMT_NF}" - while read -r pkg; do - echo -e "${FMT_GREEN} $pkg${FMT_NF}" - done <"$STATUS_DIR/upstream-pushed" - echo "" - fi - - if [[ -f "$STATUS_DIR/merge-pushed" ]]; then - echo -e "${FMT_GREEN}Merged following packages with" \ - "upstream:${FMT_NF}" - while read -r pkg; do - echo -e "${FMT_GREEN} $pkg${FMT_NF}" - done <"$STATUS_DIR/merge-pushed" - echo "" - fi - - if [[ -f "$STATUS_DIR/update-failed" ]]; then - echo -e "${FMT_RED}Failed to update following packages:" \ - "${FMT_NF}" - while read -r pkg; do - echo -e "${FMT_RED} $pkg${FMT_NF}" - done <"$STATUS_DIR/update-failed" - echo "" - fi - - if [[ -f "$STATUS_DIR/unexpected-failure" ]]; then - echo -e "${FMT_RED}Unexpected failure when updating following" \ - "packages:${FMT_NF}" - while read -r pkg; do - echo -e "${FMT_RED} $pkg${FMT_NF}" - done <"$STATUS_DIR/unexpected-failure" - echo "" - fi - - echo_bold "___________________________________________" - echo "" -} - -function report_status_all() { - report_status - without_colors report_status >"$STATUS_DIR/report" -} - -# -# This script will attempt to update every package. It will continue running -# for all the packages even if update fails for a package. -# -# The steps for updating each package are the following: -# 1) Run "buildpkg.sh -u " to update upstream, then attempt merge and -# build. If everything succeeded buildpkg.sh will return success. If only -# some parts succeeded, buildpkg.sh will record status in the package's -# WORKDIR. -# 2) If upstream was updated, push the update to the package repository. -# 3) If a merge with upstream was necessary and we succesfully performed it, -# push the merge to the package repository. -# - -if [[ -n "$UPDATE_PACKAGE_NAME" ]]; then - echo_bold "Updating only package '$UPDATE_PACKAGE_NAME' as" \ - "UPDATE_PACKAGE_NAME is set" - logmust check_package_exists "$UPDATE_PACKAGE_NAME" - PACKAGES=("$UPDATE_PACKAGE_NAME") - echo_bold "auto-mege-blacklist.pkgs is ignored" - NO_MERGE_PACKAGES=() -else - logmust read_package_list "$pkg_list_file" - PACKAGES=("${_RET_LIST[@]}") - logmust read_package_list "$TOP/package-lists/auto-merge-blacklist.pkgs" - NO_MERGE_PACKAGES=("${_RET_LIST[@]}") -fi - -for pkg in "${PACKAGES[@]}"; do - echo "" - echo_bold "Updating package $pkg." - - logmust load_package_config "$pkg" - - WORKDIR="$TOP/packages/$pkg/tmp" - unexpected_failure=false - - flags="" - for no_merge_pkg in "${NO_MERGE_PACKAGES[@]}"; do - if [[ "$pkg" == "$no_merge_pkg" ]]; then - echo_bold "Auto-merge disabled for $pkg as package" \ - "is in auto-merge-blacklist.pkgs" - flags="-M" - fi - done - - echo "Running: ./buildpkg.sh $flags -u $pkg" - if ./buildpkg.sh -u $flags "$pkg"; then - buildpkg_success=true - else - if [[ -f "$WORKDIR/updating-upstream" ]]; then - echo_error "Failed to update upstream for $pkg" - elif [[ -f "$WORKDIR/merging" ]]; then - echo_error "Failed merge with upstream for $pkg" - elif [[ -f "$WORKDIR/building" ]]; then - echo_error "Failed build of $pkg after merge" - else - die "Unexpected failure of buildpkg.sh for $pkg" - fi - - echo_error "buildpkg.sh failed." - echo "$pkg" >>"$STATUS_DIR/update-failed" - buildpkg_success=false - record_failure - fi - - if [[ -f "$WORKDIR/upstream-updated" ]]; then - # - # We push to upstream if it was updated, even if the merge with - # our repo or the build has failed. This way developers can - # manually perform the merge without having to update upstream - # themselves. - # - $dry_run && flags="-n" || flags="" - echo "Running: ./push-updates.sh -u -y $flags $pkg" - if ./push-updates.sh -u -y $flags "$pkg"; then - echo "$pkg" >>"$STATUS_DIR/upstream-pushed" - else - echo_error "Failed to push upstream changes for $pkg." - unexpected_failure=true - record_failure - fi - fi - - if ! $unexpected_failure && [[ -f "$WORKDIR/repo-updated" ]]; then - # sanity check - $buildpkg_success || - die "Repo should not be updated when buildpkg.sh failed" - - $dry_run && flags="-n" || flags="" - echo "Running ./push-updates.sh -m -y $flags $pkg" - if ./push-updates.sh -m -y $flags "$pkg"; then - echo "$pkg" >>"$STATUS_DIR/merge-pushed" - else - echo_error "Failed to push merge for $pkg." - unexpected_failure=true - record_failure - fi - fi - - if $unexpected_failure; then - echo "$pkg" >>"$STATUS_DIR/unexpected-failure" - fi - echo_bold "====================================================================" -done - -trap - EXIT - -report_status_all - -$FAILURE && exit 1 || exit 0