Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Artifactory for Python #14473

Merged
merged 25 commits into from
Dec 30, 2022
Merged

[CI] Artifactory for Python #14473

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
36e0337
Artifactory for Python
Nov 24, 2022
4a6049c
Remove setup_python_mirror for some places
Nov 28, 2022
a84d2a4
Remove setup_python_mirror for deps_build
Nov 28, 2022
3abbcee
Add setup_python_mirror
Nov 28, 2022
7855a23
Fix artifactory pypi path
Nov 28, 2022
4facbb2
Merge branch 'main' into amenasria/artifactory-for-python
Dec 13, 2022
d745e31
Add pip_config_file and python_mirror to omnibus-build task
Dec 20, 2022
ce0a77c
Use pip_config_file and python_mirror in deb rpm and suse builds
Dec 21, 2022
2291ce7
Use ARTIFACTORY_TOKEN bait to check that the CI hides it properly
Dec 21, 2022
7cf1df8
Fix typo env ENV
Dec 21, 2022
c346801
Add missing references
Dec 21, 2022
c2daefd
Pass PIP_INDEX_URL through env not to display the secret in the CI
Dec 21, 2022
5156ec1
Remove fake credentials now that CI is safe
Dec 21, 2022
a34a1f5
Add live_stream=true to have pip output
Dec 22, 2022
9f6f4f7
Revert last commit
Dec 22, 2022
da07c4d
Do the same for python 3
Dec 22, 2022
f4b85ec
Add the python mirror for pylint
Dec 23, 2022
a6e8462
Add the python mirror for windows
Dec 23, 2022
d7e13b4
Add missing tests for windows
Dec 28, 2022
299ba8c
Add debug log level for pip
Dec 28, 2022
a025aab
Trigger pipeline
Dec 28, 2022
377b214
Merge branch 'main' into amenasria/artifactory-for-python
Dec 28, 2022
a3141ba
Fix comments
Dec 29, 2022
98c297a
Remove log_level=true in the omnibus command
Dec 29, 2022
b4e1836
Missing comment fix
Dec 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,9 @@ variables:
ARTIFACTORY_TOKEN_SSM_NAME: ci.datadog-agent.artifactory_token
ARTIFACTORY_URL: datadog.jfrog.io
ARTIFACTORY_GEMS_PATH: artifactory/api/gems/agent-gems
ARTIFACTORY_PYPI_PATH: artifactory/api/pypi/agent-pypi/simple
USE_CACHING_PROXY_RUBY: "true"
USE_CACHING_PROXY_PYTHON: "true"
CLANG_LLVM_VER: 12.0.1

#
Expand Down
1 change: 1 addition & 0 deletions .gitlab/maintenance_jobs/docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ delete_docker_tag:
before_script:
- DOCKER_REGISTRY_LOGIN=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.$DOCKER_REGISTRY_LOGIN_SSM_KEY --with-decryption --query "Parameter.Value" --out text)
- PASS=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.$DOCKER_REGISTRY_PWD_SSM_KEY --with-decryption --query "Parameter.Value" --out text)
- !reference [.setup_python_mirror_linux]
- python3 -m pip install -r requirements.txt
- |
export DOCKER_TOKEN=`curl -s -H "Content-Type: application/json" -X POST -d '{"username": "'$DOCKER_REGISTRY_LOGIN'", "password": "'$PASS'"}' https://hub.docker.com/v2/users/login/ | python -c 'import sys, json; print(json.load(sys.stdin)["token"].strip())'`
Expand Down
2 changes: 2 additions & 0 deletions .gitlab/notify.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ notify:
script:
- set +x
- export GITLAB_TOKEN=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.gitlab_read_api_token --with-decryption --query "Parameter.Value" --out text)
- !reference [.setup_python_mirror_linux]
- python3 -m pip install -r tasks/libs/requirements-notifications.txt
- |
# Do not send notifications if this is a child pipeline of another repo
Expand All @@ -51,5 +52,6 @@ send_pipeline_stats:
- set +x
- export GITLAB_TOKEN=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.gitlab_read_api_token --with-decryption --query "Parameter.Value" --out text)
- export DD_API_KEY=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.datadog_api_key_org2 --with-decryption --query "Parameter.Value" --out text)
- !reference [.setup_python_mirror_linux]
- python3 -m pip install -r tasks/libs/requirements-stats.txt
- invoke -e pipeline.send-stats
1 change: 1 addition & 0 deletions .gitlab/package_build/dmg.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
- export GITHUB_KEY_B64=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.macos_github_key_b64 --with-decryption --query "Parameter.Value" --out text)
- export GITHUB_APP_ID=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.macos_github_app_id --with-decryption --query "Parameter.Value" --out text)
- export GITHUB_INSTALLATION_ID=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.macos_github_installation_id --with-decryption --query "Parameter.Value" --out text)
- !reference [.setup_python_mirror_linux]
- python3 -m pip install -r tasks/libs/requirements-github.txt
- inv -e github.trigger-macos-build --datadog-agent-ref "$CI_COMMIT_SHA" --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --destination "$OMNIBUS_PACKAGE_DIR"
- !reference [.upload_sbom_artifacts]
Expand Down
10 changes: 10 additions & 0 deletions .gitlab/shared.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,13 @@
.setup_ruby_mirror_win:
- *get_artifactory_token_win
- if ($Env:USE_CACHING_PROXY_RUBY -eq "true") { $BUNDLE_MIRROR__RUBYGEMS__ORG="https://${Env:ARTIFACTORY_USERNAME}:${ARTIFACTORY_TOKEN}@${Env:ARTIFACTORY_URL}/${Env:ARTIFACTORY_GEMS_PATH}" }

.setup_python_mirror_linux:
- set +x
- *get_artifactory_token_linux
- if [ "${USE_CACHING_PROXY_PYTHON}" = "true" ]; then export PIP_INDEX_URL=https://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_TOKEN}@${ARTIFACTORY_URL}/${ARTIFACTORY_PYPI_PATH}; fi
- set -x

.setup_python_mirror_win:
- *get_artifactory_token_win
- if ($Env:USE_CACHING_PROXY_PYTHON -eq "true") { $PIP_INDEX_URL="https://${Env:ARTIFACTORY_USERNAME}:${ARTIFACTORY_TOKEN}@${Env:ARTIFACTORY_URL}/${Env:ARTIFACTORY_PYPI_PATH}" }
1 change: 1 addition & 0 deletions .gitlab/source_test/macos.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ tests_macos:
- export GITHUB_KEY_B64=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.macos_github_key_b64 --with-decryption --query "Parameter.Value" --out text)
- export GITHUB_APP_ID=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.macos_github_app_id --with-decryption --query "Parameter.Value" --out text)
- export GITHUB_INSTALLATION_ID=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.macos_github_installation_id --with-decryption --query "Parameter.Value" --out text)
- !reference [.setup_python_mirror_linux]
- python3 -m pip install -r tasks/libs/requirements-github.txt
- inv -e github.trigger-macos-test --datadog-agent-ref "$CI_COMMIT_SHA" --python-runtimes "$PYTHON_RUNTIMES"
artifacts:
Expand Down