extensions: Separate generation of extension db from docs in CI by phlax · Pull Request #15117 · envoyproxy/envoy

phlax · 2021-02-19T15:00:06Z

Signed-off-by: Ryan Northey ryan@synca.io

Commit Message: extensions: Separate generation of extension db from docs in CI
Additional Description:

As we will require the extension DB to be generated for docs and schema, it makes sense to separate this task so that those jobs can run in parallel

Im wondering if this job should be moved to a bazel build somewhere - but initially im just gonna get it working by splitting the script/job

UPDATE

rather than separating the ci job, i have moved the db generation into a bazel job - which will allow other tasks that also require this data to make use of it more easily/bazely

this also adds buildozer into the bazel env, so we might want to make use of that elsewhere and potentially remove buildozer from the build-image env

Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue] Touch #13229
[Optional Deprecated:]
[Optional API Considerations:]

phlax · 2021-02-19T17:20:01Z

im not 100% how best to separate this, or exactly how it will be required by #15100 - so this PR is currently experimental to figure these things out

repokitteh-read-only · 2021-02-26T06:04:31Z

CC @envoyproxy/dependency-shepherds: Your approval is needed for changes made to (bazel/.*repos.*\.bzl)|(bazel/dependency_imports\.bzl)|(api/bazel/.*\.bzl)|(.*/requirements\.txt)|(.*\.patch).

🐱

Caused by: #15117 was synchronize by phlax.

see: more, trace.

phlax · 2021-02-26T18:10:53Z

@htuch would you mind having a look at this one

im not sure if its done the best way - using /source/... feels a bit hacky

also, i had to change the visibility of the actual //source and include in the bazel data - not sure if i understand why

source/extensions/extensions_build_config.bzl

tools/extensions/BUILD

Signed-off-by: Ryan Northey <ryan@synca.io>

tools/extensions/generate_extension_db.py

source/extensions/extensions_build_config.bzl

htuch · 2021-02-28T15:59:13Z

@phlax generally this is a really nice step. I don't mind hardcoding /source as the default for ENVOY_SOURCE, but maybe you want to use ENVOY_SRCDIR as done elsewhere in ci/. The visibility stuff would benefit from better understanding, since I rather keep the limitation in at least some CI checked build to avoid accidental dependencies across core/extensions happening.

repokitteh-read-only · 2021-03-01T04:34:38Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #15117 (comment) was created by @phlax.

see: more, trace.

phlax · 2021-03-01T05:08:27Z

/retest

repokitteh-read-only · 2021-03-01T05:08:33Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #15117 (comment) was created by @phlax.

see: more, trace.

phlax · 2021-03-01T05:34:05Z

im pretty sure the coverage test fail is unrelated - retesting doesnt seem to fix

phlax · 2021-03-01T13:33:10Z

BUILD

        "//test/extensions/...",
        "//test/server",
        "//test/server/config_validation",
+        "//tools/extensions/...",


maybe this should be more specific - not sure if the elipsis is necessary

I'd make is as narrow as works.

afaict this is as narrow as i can get it

Signed-off-by: Ryan Northey <ryan@synca.io>

htuch

LGTM modulo nits.
/wait

htuch · 2021-03-02T03:00:01Z

tools/extensions/generate_extension_db.py

 if __name__ == '__main__':
-  output_path = sys.argv[1]
+  try:
+    output_path = (os.environ["EXTENSION_DB_PATH"]


output_path = os.getenv('EXTENSION_DB_PATH', sys.argv[1])

this would break if sys.argv[1] is not set (ie even when EXTENSION_DB_PATH was) - hence the if ... else

i think this code could be a bit prettier with different formatting but i think the logic is correct

i updated to output_path = os.getenv("EXTENSION_DB_PATH") or sys.argv[1] which is more concise and works as expected

htuch · 2021-03-02T03:01:18Z

BUILD

        "//test/extensions/...",
        "//test/server",
        "//test/server/config_validation",
+        "//tools/extensions/...",


I'd make is as narrow as works.

Signed-off-by: Ryan Northey <ryan@synca.io>

phlax · 2021-03-02T09:51:35Z

/retest

repokitteh-read-only · 2021-03-02T09:51:39Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #15117 (comment) was created by @phlax.

see: more, trace.

htuch

LGTM, thanks!

htuch · 2021-03-02T18:47:31Z

/lgtm deps

moderation · 2021-03-02T19:20:52Z

@phlax I'll be having a stern word with @htuch on his deps LGTM 😄 The version is hardcoded in strip_prefix and urls. Updated for 4.0.1 we should have

    com_github_bazelbuild_buildtools = dict(
        project_name = "Bazel build tools",
        project_desc = "Developer tools for working with Google's bazel buildtool.",
        project_url = "https://github.com/bazelbuild/buildtools",
        version = "4.0.1",
        sha256 = "0d3ca4ed434958dda241fb129f77bd5ef0ce246250feed2d5a5470c6f29a77fa",
        release_date = "2021-03-01",
        strip_prefix = "buildtools-{version}",
        urls = ["https://github.com/bazelbuild/buildtools/archive/{version}.tar.gz"],
        use_category = ["api"],
    ),

phlax · 2021-03-02T19:25:39Z

i can add a cleanup if you want

htuch · 2021-03-02T21:12:48Z

@moderation good point; can we add a check on this kind of hardcoding to the deps validation Python?

moderation · 2021-03-02T22:47:06Z

A check in deps validation definitely makes sense. Will make an attempt.

phlax changed the title ~~[WIP] extensions: Separate generate of extension db from docs in CI~~ [WIP] extensions: Separate generation of extension db from docs in CI Feb 19, 2021

phlax marked this pull request as draft February 19, 2021 15:00

phlax force-pushed the extensions-separate-db-generation branch 3 times, most recently from 57f11dc to e41dd41 Compare February 19, 2021 17:10

phlax force-pushed the extensions-separate-db-generation branch from e41dd41 to 12485ed Compare February 26, 2021 06:04

repokitteh-read-only bot added the deps Approval required for changes to Envoy's external dependencies label Feb 26, 2021

phlax force-pushed the extensions-separate-db-generation branch 9 times, most recently from 2e06e9f to dbe4373 Compare February 26, 2021 17:58

phlax changed the title ~~[WIP] extensions: Separate generation of extension db from docs in CI~~ extensions: Separate generation of extension db from docs in CI Feb 26, 2021

phlax marked this pull request as ready for review February 26, 2021 18:09

phlax commented Feb 27, 2021

View reviewed changes

source/extensions/extensions_build_config.bzl Outdated Show resolved Hide resolved

phlax commented Feb 27, 2021

View reviewed changes

tools/extensions/BUILD Outdated Show resolved Hide resolved

phlax force-pushed the extensions-separate-db-generation branch from b4a3d0c to 1f65be9 Compare February 28, 2021 10:40

extensions: Generate extension db in bazel action

dbffcf8

Signed-off-by: Ryan Northey <ryan@synca.io>

phlax force-pushed the extensions-separate-db-generation branch from 1f65be9 to dbffcf8 Compare February 28, 2021 11:11

phlax added 2 commits February 28, 2021 11:21

docs/

ff9492d

Signed-off-by: Ryan Northey <ryan@synca.io>

docs/

f258049

Signed-off-by: Ryan Northey <ryan@synca.io>

htuch reviewed Feb 28, 2021

View reviewed changes

tools/extensions/generate_extension_db.py Outdated Show resolved Hide resolved

tools/extensions/generate_extension_db.py Outdated Show resolved Hide resolved

source/extensions/extensions_build_config.bzl Outdated Show resolved Hide resolved

phlax commented Mar 1, 2021

View reviewed changes

phlax mentioned this pull request Mar 1, 2021

coverage: adjust threshold on source/server to fix flaky failure #15238

Merged

mathetake mentioned this pull request Mar 1, 2021

coverage flakiness on source/server #15239

Closed

Merge branch 'main' into extensions-separate-db-generation

9783e0f

Signed-off-by: Ryan Northey <ryan@synca.io>

mattklein123 assigned htuch Mar 1, 2021

htuch suggested changes Mar 2, 2021

View reviewed changes

repokitteh-read-only bot added the waiting label Mar 2, 2021

build

47175d7

Signed-off-by: Ryan Northey <ryan@synca.io>

repokitteh-read-only bot removed the waiting label Mar 2, 2021

phlax added 4 commits March 2, 2021 04:37

build

f017562

Signed-off-by: Ryan Northey <ryan@synca.io>

build

6ec95ae

Signed-off-by: Ryan Northey <ryan@synca.io>

build

98348c7

Signed-off-by: Ryan Northey <ryan@synca.io>

build

2f96c1c

Signed-off-by: Ryan Northey <ryan@synca.io>

phlax requested a review from htuch March 2, 2021 05:12

tools/

5ed3634

Signed-off-by: Ryan Northey <ryan@synca.io>

htuch approved these changes Mar 2, 2021

View reviewed changes

repokitteh-read-only bot removed the deps Approval required for changes to Envoy's external dependencies label Mar 2, 2021

htuch merged commit a03209d into envoyproxy:main Mar 2, 2021

Conversation

phlax commented Feb 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

UPDATE

Uh oh!

phlax commented Feb 19, 2021

Uh oh!

repokitteh-read-only bot commented Feb 26, 2021

Uh oh!

phlax commented Feb 26, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

htuch commented Feb 28, 2021

Uh oh!

repokitteh-read-only bot commented Mar 1, 2021

Uh oh!

phlax commented Mar 1, 2021

Uh oh!

repokitteh-read-only bot commented Mar 1, 2021

Uh oh!

phlax commented Mar 1, 2021

Uh oh!

phlax Mar 1, 2021

Choose a reason for hiding this comment

Uh oh!

htuch Mar 2, 2021

Choose a reason for hiding this comment

Uh oh!

phlax Mar 2, 2021

Choose a reason for hiding this comment

Uh oh!

htuch left a comment

Choose a reason for hiding this comment

Uh oh!

htuch Mar 2, 2021

Choose a reason for hiding this comment

Uh oh!

phlax Mar 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phlax Mar 2, 2021

Choose a reason for hiding this comment

Uh oh!

htuch Mar 2, 2021

Choose a reason for hiding this comment

Uh oh!

phlax commented Mar 2, 2021

Uh oh!

repokitteh-read-only bot commented Mar 2, 2021

Uh oh!

htuch left a comment

Choose a reason for hiding this comment

Uh oh!

htuch commented Mar 2, 2021

Uh oh!

moderation commented Mar 2, 2021

Uh oh!

phlax commented Mar 2, 2021

Uh oh!

htuch commented Mar 2, 2021

Uh oh!

moderation commented Mar 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

phlax commented Feb 19, 2021 •

edited

Loading

phlax Mar 2, 2021 •

edited

Loading