-
Notifications
You must be signed in to change notification settings - Fork 196
Add buildkite pipeline for publishing to PyPI #3748
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
56 commits
Select commit
Hold shift + click to select a range
cca9bcd
Add buildkite pipeline for pypi publishing
lorenabalan a5b3b0d
Fix notice check
lorenabalan c50d937
Consolidate tools config in pyproject.toml
lorenabalan b7ca75d
Blank line
lorenabalan 27f1377
Add manual confirmation step for now
lorenabalan 4dd68c5
Update manual release pipeline
lorenabalan 1db805a
Update current pipeline
lorenabalan f06b27b
Restore notice
lorenabalan f4b96cf
Add missing package_path
lorenabalan bc9d3e2
Update NOTICE.txt
elasticmachine aac7627
Merge branch 'lb/pypi-pipeline' of github.com:elastic/connectors into…
lorenabalan 3ad00aa
Fix lint
lorenabalan ab30eb8
Run package check quicker
lorenabalan d41045c
Add license as per PEP621
lorenabalan 43edc13
Add connectors* to packages.find
lorenabalan 34f2b71
Update NOTICE.txt
elasticmachine f2a63ce
Update license
lorenabalan eb5072a
Make build check work on Python 3.11
lorenabalan 8036357
Update NOTICE.txt
elasticmachine 97b52fd
Merge branch 'develop' into lb/pypi-pipeline
lorenabalan ee069dd
Merge branch 'lb/pypi-pipeline' of github.com:elastic/connectors into…
lorenabalan 5b5c13b
Fixxxxx notice
lorenabalan 8619c1e
Update NOTICE.txt
elasticmachine 23f7632
Correct buildkite label
lorenabalan d7f7a96
Use script in pypi-publish pipeline too
lorenabalan ccfd0f3
Update NOTICE.txt
elasticmachine a659961
Use script to publish to PyPIs
lorenabalan 60b2cd5
Merge branch 'lb/pypi-pipeline' of github.com:elastic/connectors into…
lorenabalan fa92b6b
Add new buildkite pipeline to catalog
lorenabalan 53c130a
Typo
lorenabalan 915bb5a
Test CLI
lorenabalan 684d3d3
Fix CLI
lorenabalan 5156e7f
Fix tests CLI
lorenabalan 8c02e68
Install from binary distribution instead of from local source
lorenabalan ae00648
Update NOTICE.txt
elasticmachine da28bf4
Publish only one whl
lorenabalan 1035993
Blank line
lorenabalan feff713
Merge branch 'develop' into lb/pypi-pipeline
lorenabalan af056ad
Update NOTICE.txt
elasticmachine cdc9e7a
Publish binary and source distributions as DRAs
lorenabalan 949d528
Merge branch 'lb/pypi-pipeline' of github.com:elastic/connectors into…
lorenabalan 671ddf9
Rename
lorenabalan e8f92d3
Fix ftests
lorenabalan 92dae7c
Comment out real pypi publishing
lorenabalan ec4d050
Rename step, update Dockerfile
lorenabalan ee2d136
Update version
lorenabalan ad1e6e2
Update NOTICE.txt
elasticmachine f7ec4f6
Merge branch 'develop' into lb/pypi-pipeline
lorenabalan 51d3e46
Update pyproject.toml
lorenabalan 6a2d1b2
Update license
lorenabalan d04d9c6
Update NOTICE.txt
elasticmachine 4d185fe
Add search-eng team to bk pipeline
lorenabalan 072855a
Use DRA_ARTIFACTS_DIR everywhere
lorenabalan df3f4d6
Update Vault path
lorenabalan e27a7ae
Small package test
lorenabalan 2024377
pedantic reorg of cp statements (#3813)
seanstory File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| #!/bin/bash | ||
|
|
||
| # !!! WARNING DO NOT add -x to avoid leaking vault passwords | ||
| set -euo pipefail | ||
|
|
||
| source .buildkite/shared.sh | ||
|
|
||
| init_python | ||
| cd "$PACKAGE_PATH" | ||
| python -m pip install --upgrade build twine | ||
| python -m build | ||
|
|
||
| export TWINE_USERNAME="__token__" | ||
|
|
||
| # upload to test or real PyPI based on TEST_PYPI=1 env var or arg | ||
| if [[ "${1:-}" == "TEST_PYPI=1" ]] || [[ "${TEST_PYPI:-}" =~ ^(1|TRUE|true)$ ]]; then | ||
| TWINE_PASSWORD=$(vault read -field publishing-api-key secret/ci/elastic-connectors/test-pypi) | ||
| export TWINE_PASSWORD | ||
| python -m twine upload --repository testpypi dist/* | ||
| else | ||
| TWINE_PASSWORD=$(vault read -field publishing-api-key secret/ci/elastic-connectors/pypi) | ||
| export TWINE_PASSWORD | ||
| python -m twine upload --repository pypi dist/* | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| ## .buildkite/pypi-publish-pipeline.yml | ||
| # Manually-triggered pipeline to build and publish Python packages to PyPI | ||
| notify: | ||
| - if: 'build.branch =~ /^((main)|([0-9]+\.[0-9]+))$/ && (build.state == "failed" || pipeline.started_passing)' | ||
| slack: | ||
| channels: | ||
| - "#search-et-alerts" | ||
| message: "${BUILDKITE_MESSAGE}" | ||
|
|
||
| # add paths for all the packages we want to build here | ||
| # then make sure they're added to the matrix.setup.package_path lists in the steps below | ||
| env: | ||
| SERVICE_PACKAGE_PATH: "app/connectors_service" | ||
| SDK_PACKAGE_PATH: "libs/connectors_sdk" | ||
|
|
||
| steps: | ||
| - group: ":building_construction: Build Python Packages" | ||
| key: "build_package" | ||
| steps: | ||
| - label: ":python: Build Python {{matrix.python_version}} {{matrix.package_path}} package" | ||
| key: build_python_package | ||
| agents: | ||
| provider: "gcp" | ||
| machineType: "n1-standard-8" | ||
| useVault: true | ||
| image: family/enterprise-search-ubuntu-2204-connectors-py | ||
| matrix: | ||
| setup: | ||
| python_version: | ||
| - "3.10" | ||
lorenabalan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - "3.11" | ||
| package_path: | ||
| - "${SERVICE_PACKAGE_PATH}" | ||
| - "${SDK_PACKAGE_PATH}" | ||
| env: | ||
| PYTHON_VERSION: "{{ matrix.python_version }}" | ||
| PACKAGE_PATH: "{{ matrix.package_path }}" | ||
| commands: ".buildkite/test_python_packages.sh" | ||
|
|
||
| - wait | ||
|
|
||
| - group: ":test_tube: Publish to Test PyPI" | ||
| key: "publish_to_test_pypi" | ||
| depends_on: | ||
| - build_package | ||
| steps: | ||
| - label: ":package: Publish to Test PyPI" | ||
| key: publish_test_pypi | ||
| agents: | ||
| provider: "gcp" | ||
| machineType: "n1-standard-8" | ||
| useVault: true | ||
| image: family/enterprise-search-ubuntu-2204-connectors-py | ||
| matrix: | ||
| setup: | ||
| package_path: | ||
| - "${SERVICE_PACKAGE_PATH}" | ||
| - "${SDK_PACKAGE_PATH}" | ||
| env: | ||
| PYTHON_VERSION: "3.11" | ||
| PACKAGE_PATH: "{{ matrix.package_path }}" | ||
| commands: "TEST_PYPI=1 ./buildkite/publish_to_pypi.sh" | ||
|
|
||
| # FOR WHEN WE'RE READY TO PUBLISH TO PRODUCTION PYPI | ||
seanstory marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # | ||
| # - block: ":rocket: Publish to Production PyPI?" | ||
| # prompt: "Test PyPI publish succeeded. Proceed to publish to Production PyPI?" | ||
| # key: "confirm_production_publish" | ||
| # depends_on: | ||
| # - publish_to_test_pypi | ||
| # | ||
| # - group: ":truck: Publish to Production PyPI" | ||
| # key: "publish_to_pypi" | ||
| # depends_on: | ||
| # - confirm_production_publish | ||
| # steps: | ||
| # - label: ":package: Publish to Production PyPI" | ||
| # key: publish_pypi | ||
| # agents: | ||
| # provider: "gcp" | ||
| # machineType: "n1-standard-8" | ||
| # useVault: true | ||
| # image: family/enterprise-search-ubuntu-2204-connectors-py | ||
| # matrix: | ||
| # setup: | ||
| # package_path: | ||
| # - "${SERVICE_PACKAGE_PATH}" | ||
| # - "${SDK_PACKAGE_PATH}" | ||
| # env: | ||
| # PYTHON_VERSION: "3.11" | ||
| # PACKAGE_PATH: "{{ matrix.package_path }}" | ||
| # commands: ".buildkite/publish_to_pypi.sh" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| #!/bin/bash | ||
|
|
||
| # !!! WARNING DO NOT add -x to avoid leaking vault passwords | ||
| set -euo pipefail | ||
|
|
||
| source .buildkite/shared.sh | ||
|
|
||
| init_python | ||
|
|
||
| python -m pip install --upgrade build twine | ||
| python -m build "$PACKAGE_PATH" | ||
| ls -lah "$PACKAGE_PATH/dist/" | ||
| python -m twine check "$PACKAGE_PATH/dist/*" | ||
|
|
||
| # If this is the connectors_service package, test the installation and CLI | ||
| if [[ "$PACKAGE_PATH" == *app/connectors_service* ]]; then | ||
lorenabalan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| echo "Testing connectors_service package installation and CLI..." | ||
|
|
||
| # Install the connectors_sdk package first | ||
| LIB_PATH="libs/connectors_sdk" | ||
| python -m build "$LIB_PATH" | ||
| python -m pip install "$LIB_PATH"/dist/*.whl | ||
|
|
||
| python -m pip install "$PACKAGE_PATH"/dist/*.whl | ||
| connectors --help | ||
| elastic-ingest --help | ||
| # elastic-agent-connectors --help | ||
| test-connectors --help | ||
lorenabalan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| else | ||
| python -m pip install "$PACKAGE_PATH"/dist/*.whl | ||
| python -c "import connectors_sdk; print(f'🎉 Success! connectors_sdk version: {connectors_sdk.__version__}')" | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.