Add check of component versions to CI

.github/workflows/manifests.yml

@@ -3,6 +3,8 @@ name: manifests
 on:
   push:
   pull_request:
+    paths:
+      - manifests/*.yml
   schedule:
     - cron: "0 0 * * *"
 
@@ -22,11 +24,17 @@ jobs:
     runs-on: ubuntu-latest
     env:
       PYTHON_VERSION: 3.7
+      JDK_VERSION: 14
     strategy:
       matrix:
         manifest: ${{ fromJson(needs.list-manifests.outputs.matrix) }}
     steps:
+
       - uses: actions/checkout@v2
+      - name: Set Up JDK ${{ env.JDK_VERSION }}
+        uses: actions/setup-java@v1
+        with:
+          java-version: ${{ env.JDK_VERSION }}
       - name: Set up Python ${{ env.PYTHON_VERSION }}
         uses: actions/setup-python@v2
         with:
@@ -36,4 +44,4 @@ jobs:
           python -m pip install --upgrade pipenv wheel
       - name: OpenSearch Manifests
         run: |
-          ./bundle-workflow/ci.sh ${{ matrix.manifest }}
+          ./bundle-workflow/ci.sh ${{ matrix.manifest }} --snapshot

DEVELOPER_GUIDE.md

@@ -141,7 +141,7 @@ Run from `bundle-workflow` before making pull requests.
 cd bundle-workflow
 
 pipenv run isort .
-pipenv run black .
+git status -s | grep -e "[MA?]\s.*.py" | cut -c4- | xargs pipenv run black
 pipenv run flake8
 pipenv run pytest
 pipenv run mypy .

bundle-workflow/README.md

@@ -7,6 +7,7 @@
   - [Test the Bundle](#test-the-bundle)
     - [Integration Tests](#integration-tests)
     - [Backwards Compatibility Tests](#backwards-compatibility-tests)
+    - [Performance Tests](#performance-tests)
   - [Sanity Check the Bundle](#sanity-check-the-bundle)
 
 ## OpenSearch Bundle Workflow
@@ -15,16 +16,15 @@ This workflow builds a complete OpenSearch bundle from source. You can currently
 
 ### Build from Source
 
-Each build requires a manifest to be passed as input.  We currently have the following input manifests:
+Each build requires a manifest to be passed as input. We currently have the following input manifests.
 
-| name        | description                                                                         |
-|-------------|-------------------------------------------------------------------------------------|
-| [opensearch-1.0.0.yml](/manifests/opensearch-1.0.0.yml) |  Manifest to reproduce 1.0.0 build.                 |
-| [opensearch-1.0.0-maven.yml](/manifests/opensearch-1.0.0-maven.yml)|   One-time manifest to build and push maven artifacts for 1.0 from tags. Going forward a separate maven manifest is not required. For 1.0.0 we do not have solid 1.0 refs for all repos nor do we need to rebuild the full bundle.|
-| [opensearch-1.1.0.yml](/manifests/opensearch-1.1.0.yml)| Manifest to build upcoming 1.x release.
+| name                                                                  | description                                                   |
+|-----------------------------------------------------------------------|---------------------------------------------------------------|
+| [opensearch-1.0.0.yml](/manifests/opensearch-1.0.0.yml)               | Manifest to reproduce 1.0.0 build.                            |
+| [opensearch-1.0.0-maven.yml](/manifests/opensearch-1.0.0-maven.yml)   | One-time manifest to build maven artifacts for 1.0 from tags. |
+| [opensearch-1.1.0.yml](/manifests/opensearch-1.1.0.yml)               | Manifest to build upcoming 1.x release.                       |
 
-
-Usage:
+The following example builds a shapshot version of OpenSearch 1.1.0.
 
 ```bash
 ./bundle-workflow/build.sh manifests/opensearch-1.1.0.yml --snapshot
@@ -78,19 +78,20 @@ You can perform additional plugin install steps by adding an `install.sh` script
 
 ### Signing Artifacts
 
-The signing step (optional) takes the manifest file created from the build step and signs all its component artifacts using the opensearch-signer-client. The input requires a path to the build manifest and is expected to be inside the artifacts directory with the same directories mentioned in the build step. 
+The signing step (optional) takes the manifest file created from the build step and signs all its component artifacts using a tool called `opensearch-signer-client` (in progress of being open-sourced). The input requires a path to the build manifest and is expected to be inside the artifacts directory with the same directories mentioned in the build step. 
 
 The following options are available. 
 
-| name          | description                                                                         |
-|---------------|-------------------------------------------------------------------------------------|
-| --component   | The component name of the component whose artifacts will be signed                  |
-| --type        | The artifact type to be signed. Currently one of 3 options: [plugins, maven, bundle]|
-| -v, --verbose | Show more verbose output.                                                           |
+| name          | description                                                                           |
+|---------------|---------------------------------------------------------------------------------------|
+| --component   | The component name of the component whose artifacts will be signed.                   |
+| --type        | The artifact type to be signed. Currently one of 3 options: [plugins, maven, bundle]. |
+| -v, --verbose | Show more verbose output.                                                             |
 
 The signed artifacts (<artifact>.asc) will be found in the same location as the original artifact. 
 
-Signing step (to sign all artifacts):
+The following command signs all artifacts.
+
 ```bash
 ./bundle_workflow/sign.sh artifacts/manifest.yml
 ```
@@ -99,21 +100,12 @@ Signing step (to sign all artifacts):
 
 Tests the OpenSearch bundle.
 
-This workflow contains two sections: Integration Tests, Backwards Compatibility Tests.
-
-#### Integration Tests
-
-This step runs integration tests invoking `integtest.sh` in each component from bundle manifest.
-
-#### Backwards Compatibility Tests
-
-This step run backward compatibility invoking `bwctest.sh` in each component from bundle manifest.
+This workflow contains integration, backwards compatibility and performance tests.
 
-Usage:
+The following example kicks off all test suites for a distribution of OpenSearch 1.1.0.
 
-Kick off all test suites on a manifest:
 ```bash
-./bundle-workflow/test.sh manifests/opensearch-1.0.0.yml
+./bundle-workflow/test.sh manifests/opensearch-1.1.0.yml
 ```
 
 The following options are available.
@@ -124,16 +116,46 @@ The following options are available.
 | --keep             | Do not delete the temporary working directory on both success or error. |
 | -v, --verbose      | Show more verbose output.                                               |
 
+#### Integration Tests
+
+This step runs integration tests invoking `integtest.sh` in each component from bundle manifest.
+
+#### Backwards Compatibility Tests
+
+This step run backward compatibility invoking `bwctest.sh` in each component from bundle manifest.
+
+#### Performance Tests
+
+TODO
+
 ### Sanity Check the Bundle
 
-Runs basic sanity checks on the OpenSearch bundle.
+This workflow runs sanity checks on every component present in the bundle, executed as part of the [manifests workflow](/.github/workflows/manifests.yml) in this repostiory. It ensures that the component GitHub repositories are correct and versions in those components match the OpenSearch version.
+
+To use checks, nest them under `checks` in the manifest.
+
+```yaml
+- name: common-utils
+  repository: https://github.com/opensearch-project/common-utils.git
+  ref: main
+  checks:
+    - gradle:publish
+    - gradle:properties:version
+```
+
+The following checks are available.
 
-This workflow runs basic sanity checks on every component present in the bundle. For starters, it ensures that the component GitHub repositories are correct.
+| name                                          | description                                                   |
+|-----------------------------------------------|---------------------------------------------------------------|
+| gradle:properties:version                     | Check version of the component.                               |
+| gradle:dependencies:opensearch.version        | Check dependency on the correct version of OpenSearch.        |
+| gradle:plugin.dependencies:opensearch.version | Check plugin dependency on the correct version of OpenSearch. |                                              |
+| gradle:publish                                | Check that publishing to Maven local works, and publish.      |
 
-Usage:
+The following example sanity-checks components in the the OpenSearch 1.1.0 manifest.
 
 ```bash
-./bundle-workflow/ci.sh manifests/opensearch-1.1.0.yml
+./bundle-workflow/ci.sh manifests/opensearch-1.1.0.yml --snapshot
 ```
 
 The following options are available.

bundle-workflow/src/build_workflow/build_recorder.py

@@ -10,59 +10,16 @@
 from zipfile import ZipFile
 
 import yaml
-from jproperties import Properties  # type: ignore
 
 from manifests.build_manifest import BuildManifest
+from system.properties_file import PropertiesFile
 
 
 class BuildRecorder:
     class ArtifactInvalidError(Exception):
         def __init__(self, path, message):
             self.path = path
-            super().__init__(
-                f"Artifact {os.path.basename(path)} is invalid: {message}."
-            )
-
-    class PropertiesFile(Properties):
-        def __init__(self, filename, data):
-            super().__init__(self)
-            self.filename = filename
-            self.load(data)
-
-        def get_value(self, key, default_value=None):
-            try:
-                return self[key].data
-            except KeyError:
-                return default_value
-
-        def check_value(self, key, expected_value):
-            try:
-                value = self[key].data
-                if value != expected_value:
-                    raise BuildRecorder.ArtifactInvalidError(
-                        self.filename,
-                        f"expected to have {key}={expected_value}, but was {value}",
-                    )
-            except KeyError:
-                raise BuildRecorder.ArtifactInvalidError(
-                    self.filename,
-                    f"expected to have {key}={expected_value}, but none was found",
-                )
-
-        def check_value_in(self, key, expected_values):
-            try:
-                value = self[key].data
-                if value not in expected_values:
-                    raise BuildRecorder.ArtifactInvalidError(
-                        self.filename,
-                        f"expected to have {key}=any of {expected_values}, but was {value}",
-                    )
-            except KeyError:
-                if None not in expected_values:
-                    raise BuildRecorder.ArtifactInvalidError(
-                        self.filename,
-                        f"expected to have {key}=any of {expected_values}, but none was found",
-                    )
+            super().__init__(f"Artifact {os.path.basename(path)} is invalid. {message}")
 
     def __init__(self, build_id, output_dir, name, version, arch, snapshot):
         self.output_dir = output_dir
@@ -123,16 +80,19 @@ def __check_artifact(self, artifact_type, artifact_file):
 
     def __check_plugin_artifact(self, artifact_file):
         if os.path.splitext(artifact_file)[1] != ".zip":
-            raise BuildRecorder.ArtifactInvalidError(artifact_file, "not a zip file")
+            raise BuildRecorder.ArtifactInvalidError(artifact_file, "Not a zip file.")
         if not artifact_file.endswith(f"-{self.component_version}.zip"):
             raise BuildRecorder.ArtifactInvalidError(
-                artifact_file, f"expected filename to include {self.component_version}"
+                artifact_file, f"Expected filename to include {self.component_version}."
             )
         with ZipFile(artifact_file, "r") as zip:
             data = zip.read("plugin-descriptor.properties").decode("UTF-8")
-            properties = BuildRecorder.PropertiesFile(artifact_file, data)
-            properties.check_value("version", self.component_version)
-            properties.check_value("opensearch.version", self.version)
+            properties = PropertiesFile(data)
+            try:
+                properties.check_value("version", self.component_version)
+                properties.check_value("opensearch.version", self.version)
+            except PropertiesFile.CheckError as e:
+                raise BuildRecorder.ArtifactInvalidError(artifact_file, e.__str__())
             logging.info(
                 f'Checked {artifact_file} ({properties.get_value("version", "N/A")})'
             )
@@ -159,11 +119,14 @@ def __check_maven_artifact(self, artifact_file):
         if os.path.splitext(artifact_file)[1] == ".jar":
             with ZipFile(artifact_file, "r") as zip:
                 data = zip.read("META-INF/MANIFEST.MF").decode("UTF-8")
-                properties = BuildRecorder.PropertiesFile(artifact_file, data)
-                properties.check_value_in(
-                    "Implementation-Version",
-                    [self.component_version, self.opensearch_version, None],
-                )
+                properties = PropertiesFile(data)
+                try:
+                    properties.check_value_in(
+                        "Implementation-Version",
+                        [self.component_version, self.opensearch_version, None],
+                    )
+                except PropertiesFile.CheckError as e:
+                    raise BuildRecorder.ArtifactInvalidError(artifact_file, e.__str__())
                 logging.info(
                     f'Checked {artifact_file} ({properties.get_value("Implementation-Version", "N/A")})'
                 )

bundle-workflow/src/ci.py

@@ -37,11 +37,14 @@
 
         logging.info(f"Sanity checking {component.name}")
         repo = GitRepository(
-            component.repository, component.ref, os.path.join(work_dir, component.name)
+            component.repository,
+            component.ref,
+            os.path.join(work_dir, component.name),
+            component.working_directory,
         )
 
         try:
-            ci = Ci(component.name, repo)
+            ci = Ci(component, repo)
             ci.check(manifest.build.version, arch, args.snapshot)
         except:
             logging.error(

bundle-workflow/src/ci_workflow/check.py

@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+from abc import ABC, abstractmethod
+
+
+class Check(ABC):
+    def __init__(self, component, git_repo, version, arch, snapshot):
+        self.component = component
+        self.git_repo = git_repo
+        self.version = version
+        self.arch = arch
+        self.snapshot = snapshot
+        self.opensearch_version = version + "-SNAPSHOT" if snapshot else version
+        self.component_version = version + ".0-SNAPSHOT" if snapshot else f"{version}.0"
+        if self.component.name == "OpenSearch":
+            # HACK: OpenSearch version is 3-digits
+            self.component_version = self.opensearch_version
+
+    @abstractmethod
+    def check(self):
+        pass

bundle-workflow/src/ci_workflow/check_gradle_dependencies.py

@@ -0,0 +1,55 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+import logging
+import re
+
+from ci_workflow.check import Check
+from system.properties_file import PropertiesFile
+
+
+class CheckGradleDependencies(Check):
+    def __init__(
+        self, component, git_repo, version, arch, snapshot, gradle_project=None
+    ):
+        super().__init__(component, git_repo, version, arch, snapshot)
+        self.gradle_project = gradle_project
+        self.dependencies = self.__get_dependencies()
+
+    def __get_dependencies(self):
+        cmd = " ".join(
+            [
+                f"./gradlew {self.gradle_project or ''}:dependencies",
+                f"-Dopensearch.version={self.opensearch_version}",
+                f"-Dbuild.snapshot={str(self.snapshot).lower()}",
+                '| grep -e "---"',
+            ]
+        )
+
+        lines = self.git_repo.output(cmd)
+        stack = ["root"]
+        props = PropertiesFile("")
+        for line in lines.split("\n"):
+            # e.g. "|    +--- org.opensearch:opensearch-core:1.1.0-SNAPSHOT"
+            # see job_scheduler_dependencies.txt in tests for an example
+            match = re.search(r"---\s(.*):([0-9,\w,.-]*)[\s]*", line)
+            if match:
+                levels = line.count("   ") + line.count("---")
+
+                while levels < len(stack):
+                    del stack[-1]
+
+                if levels == len(stack):
+                    stack[-1] = match.group(1).strip()
+                elif levels > len(stack):
+                    stack.append(match.group(1).strip())
+
+                key = "/".join(stack)
+                value = match.group(2).strip()
+                logging.debug(f"{key}={value}")
+                props[key] = value
+
+        return props