automate the ironbank generation #8537
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request does not have a backport label. Could you fix it @v1v? 🙏
NOTE: |
mergify
bot
added
the
backport-skip
|
/package |
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
🤖 GitHub commentsTo re-run your PR in the CI, just comment with:
|
🌐 Coverage report
|
|
@v1v The implementation looks pretty sane. EDIT: |
v1v
commented
Jul 5, 2022
v1v
commented
Jul 5, 2022
Comment on lines
+42
to
+46
| - filename: "apm-server-{{ beat_version }}-linux-x86_64.tar.gz" | ||
| url: "<artifact_path>/apm-server-{{ beat_version }}-linux-x86_64.tar.gz" | ||
| validation: | ||
| type: "sha512" | ||
| value: "<insert SHA 512 here>" |
There was a problem hiding this comment.
This is the agreement with the Unified Release automation.
v1v
added
backport-8.3
mergify
bot
removed
the
backport-skip
📚 Go benchmark reportDiff with the report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat |
3 tasks
kuisathaverat
approved these changes
Jul 6, 2022
v1v
commented
Jul 6, 2022
.gitignore
Outdated
| @@ -53,3 +53,6 @@ terraform.tfstate.backup | |||
| terraform.tfvars | |||
| testing/benchmark/.envrc | |||
| testing/benchmark/benchmark-result.txt | |||
|
|
|||
| # Ironbank | |||
| apm-server-ironbank* | |||
There was a problem hiding this comment.
As long as the Tar compress the given folder and nested folders, then it's required to run tar within the top-level folder so the folder name contains the docker context, otherwise it will be not compliance with the Ironbank automation
There was a problem hiding this comment.
Sorry, having trouble understanding this. What is the requirement? I'm not keen on build files being created in the top level directory. Why can't the apm-server-ironbank-<version>-docker-build-context directory be created under build/?
There was a problem hiding this comment.
I tried initially that approach but the generated apm-server-ironbank-<version>-docker-build-context.tar.gz included the build/... folders. Unfortunately I could not find the way to chdir to build so the tar command could be created from that context instead.
I might need to look for how to solve this, though if you have some suggestions please let me know.
There was a problem hiding this comment.
diff --git a/magefile.go b/magefile.go
index e314dfb5b..7cd8d9d4d 100644
--- a/magefile.go
+++ b/magefile.go
@@ -261,7 +261,8 @@ func saveIronbank() error {
fmt.Println(">> saveIronbank: save the IronBank container context.")
ironbank := getIronbankContextName()
- if _, err := os.Stat(ironbank); os.IsNotExist(err) {
+ buildDir := filepath.Join("build", ironbank)
+ if _, err := os.Stat(buildDir); os.IsNotExist(err) {
return fmt.Errorf("cannot find the folder with the ironbank context")
}
@@ -276,7 +277,7 @@ func saveIronbank() error {
tarGzFile := filepath.Join(distributionsDir, ironbank+".tar.gz")
// Save the build context as tar.gz artifact
- err := mage.Tar(ironbank, tarGzFile)
+ err := mage.Tar(buildDir, tarGzFile)
if err != nil {
return fmt.Errorf("cannot compress the tar.gz file")
}
@@ -297,6 +298,7 @@ func getIronbankContextName() string {
func prepareIronbankBuild() error {
fmt.Println(">> prepareIronbankBuild: prepare the IronBank container context.")
ironbank := getIronbankContextName()
+ buildDir := filepath.Join("build", ironbank)
templatesDir := filepath.Join("packaging", "ironbank")
data := map[string]interface{}{
@@ -306,7 +308,7 @@ func prepareIronbankBuild() error {
err := filepath.Walk(templatesDir, func(path string, info os.FileInfo, _ error) error {
if !info.IsDir() {
target := strings.TrimSuffix(
- filepath.Join(ironbank, filepath.Base(path)),
+ filepath.Join(buildDir, filepath.Base(path)),
".tmpl",
)but it produces:
$ ls -ltra build
total 0
drwxr-xr-x 6 vmartinez staff 192 Jul 18 14:08 apm-server-ironbank-8.4.0-docker-build-context
drwxr-xr-x 72 vmartinez staff 2304 Jul 18 19:04 ..
drwxr-x--- 5 vmartinez staff 160 Jul 18 19:04 distributions
drwxr-x--- 4 vmartinez staff 128 Jul 18 19:04 .
$ ls -ltra build/distributions
total 24
-rw-r--r-- 1 vmartinez staff 8872 Jul 18 14:08 apm-server-ironbank-8.4.0-docker-build-context.tar.gz
-rw-r--r-- 1 vmartinez staff 183 Jul 18 14:08 apm-server-ironbank-8.4.0-docker-build-context.tar.gz.sha512
$ cd build/distributions
$ tar xvzf apm-server-ironbank-8.4.0-docker-build-context.tar.gz
x build/apm-server-ironbank-8.4.0-docker-build-context
x build/apm-server-ironbank-8.4.0-docker-build-context/Dockerfile
x build/apm-server-ironbank-8.4.0-docker-build-context/LICENSE
x build/apm-server-ironbank-8.4.0-docker-build-context/README.md
x build/apm-server-ironbank-8.4.0-docker-build-context/hardening_manifest.yaml
But I'd expect:
$ tar xvzf apm-server-ironbank-8.4.0-docker-build-context.tar.gz
x apm-server-ironbank-8.4.0-docker-build-context
x apm-server-ironbank-8.4.0-docker-build-context/Dockerfile
x apm-server-ironbank-8.4.0-docker-build-context/LICENSE
x apm-server-ironbank-8.4.0-docker-build-context/README.md
x apm-server-ironbank-8.4.0-docker-build-context/hardening_manifest.yamlThere was a problem hiding this comment.
@v1v should it even have apm-server-ironbank-8.4.0-docker-build-context/ in the contents? This is what the 8.3.2 Elasticsearch IronBank build context tarball looks like:
$ tar tf elasticsearch-ironbank-8.3.2-docker-build-context.tar.gz
scripts/
scripts/docker-openjdk
scripts/docker-entrypoint.sh
scripts/elasticsearch.yml
README.md
LICENSE
hardening_manifest.yaml
Dockerfile
scripts/log4j2.properties
There was a problem hiding this comment.
Good point, I thought I did review the folder layout for some other projects and it seems I messed up with the extra folder :/
tar -xvf kibana-ironbank-8.3.3-SNAPSHOT-docker-build-context.tar.gz
x Dockerfile
x LICENSE
x README.md
x bin/
x bin/kibana-docker
x config/
x config/kibana.yml
x hardening_manifest.yamlThere was a problem hiding this comment.
I'd say mage.Tar cannot be used as is, I'll transition this PR to draft so I'll make the changes to support the tar.gz without any nested folders
There was a problem hiding this comment.
I think I managed to solve the issue with os.Chdir so the tar.gz contains now the right files without any nested folders :)
cachedout
approved these changes
Jul 7, 2022
2 tasks
marclop
reviewed
Jul 8, 2022
|
@marclop Could you please give this one last look? Thanks. |
|
/test |
axw
approved these changes
Jul 20, 2022
Co-authored-by: Andrew Wilkins <[email protected]>
mergify bot
pushed a commit
that referenced
this pull request
Jul 20, 2022
(cherry picked from commit 682a0e5) # Conflicts: # Makefile # magefile.go
mergify bot
pushed a commit
that referenced
this pull request
Jul 20, 2022
(cherry picked from commit 682a0e5) # Conflicts: # Makefile # magefile.go
v1v
pushed a commit
that referenced
this pull request
Jul 20, 2022
v1v
pushed a commit
that referenced
this pull request
Jul 20, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation/summary
Automate the docker context generation for the IronBank releases, this will allow us to move away from creating those docker context manually in a different repository, since the Platform Release team already provide the automation.
Implementation details
A similar approach was done for Kibana and Elasticsearch
I initially enabled to build and save the docker image but it failed with some auth issues:
I don't know if other teams are actually building those docker.
DoD build context content
As required the build context contains the following files:
Dockerfilewith specific requirements for DoD (registry args, no internet dependencies, healthcheck, ...)hardening_manifest.yamlwith all required dependenciesLICENSEspecific for DoDREADME.mdwith specific content for DoDconfig/(optional) directory containing all config files to include into the Docker imagescripts/directory containing all scripts to include into the Docker image (example: entrypoint...)Tasks
x86_64.Test
This job produces the artifacts.
Further details
This could be potentially added to the Beats project, but I don't know what's the plan for the project and what's the priorities for the Platform Productivity team, but for now I think we could try to use this in the
apm-serverrepo