Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure Renovate #5056

Merged
merged 18 commits into from
Feb 2, 2021
Merged

Conversation

renovate-bot
Copy link
Contributor

@renovate-bot renovate-bot commented Jan 29, 2021

WhiteSource Renovate

Welcome to Renovate! This is an onboarding PR to help you understand and configure settings before regular Pull Requests begin.

🚦 To activate Renovate, merge this Pull Request. To disable Renovate, simply close this Pull Request unmerged.


Detected Package Files

  • WORKSPACE (bazel)
  • backend/Dockerfile (dockerfile)
  • backend/Dockerfile.bazel (dockerfile)
  • backend/Dockerfile.cacheserver (dockerfile)
  • backend/Dockerfile.persistenceagent (dockerfile)
  • backend/Dockerfile.scheduledworkflow (dockerfile)
  • backend/Dockerfile.viewercontroller (dockerfile)
  • backend/Dockerfile.visualization (dockerfile)
  • backend/metadata_writer/Dockerfile (dockerfile)
  • backend/src/cache/deployer/Dockerfile (dockerfile)
  • components/gcp/container/Dockerfile (dockerfile)
  • components/kubeflow/deployer/Dockerfile (dockerfile)
  • components/kubeflow/dnntrainer/Dockerfile (dockerfile)
  • components/kubeflow/katib-launcher/Dockerfile (dockerfile)
  • components/kubeflow/kfserving/Dockerfile (dockerfile)
  • components/kubeflow/launcher/Dockerfile (dockerfile)
  • components/local/base/Dockerfile (dockerfile)
  • components/local/confusion_matrix/Dockerfile (dockerfile)
  • components/local/roc/Dockerfile (dockerfile)
  • components/sample/keras/train_classifier/Dockerfile (dockerfile)
  • contrib/components/openvino/model_convert/containers/Dockerfile (dockerfile)
  • contrib/components/openvino/ovms-deployer/containers/Dockerfile (dockerfile)
  • contrib/components/openvino/predict/containers/Dockerfile (dockerfile)
  • contrib/components/openvino/tf-slim/containers/Dockerfile (dockerfile)
  • frontend/Dockerfile (dockerfile)
  • manifests/gcp_marketplace/deployer/Dockerfile (dockerfile)
  • proxy/Dockerfile (dockerfile)
  • samples/contrib/image-captioning-gcp/src/Dockerfile (dockerfile)
  • samples/contrib/nvidia-resnet/components/inference_server_launcher/Dockerfile (dockerfile)
  • samples/contrib/nvidia-resnet/components/preprocess/Dockerfile (dockerfile)
  • samples/contrib/nvidia-resnet/components/train/Dockerfile (dockerfile)
  • samples/contrib/nvidia-resnet/components/webapp/Dockerfile (dockerfile)
  • samples/contrib/nvidia-resnet/components/webapp_launcher/Dockerfile (dockerfile)
  • samples/contrib/nvidia-resnet/pipeline/Dockerfile (dockerfile)
  • samples/contrib/versioned-pipeline-ci-samples/helloworld-ci-sample/helloworld/Dockerfile (dockerfile)
  • samples/contrib/versioned-pipeline-ci-samples/kaggle-ci-sample/download_dataset/Dockerfile (dockerfile)
  • samples/contrib/versioned-pipeline-ci-samples/kaggle-ci-sample/submit_result/Dockerfile (dockerfile)
  • samples/contrib/versioned-pipeline-ci-samples/kaggle-ci-sample/train_model/Dockerfile (dockerfile)
  • samples/contrib/versioned-pipeline-ci-samples/kaggle-ci-sample/visualize_html/Dockerfile (dockerfile)
  • samples/contrib/versioned-pipeline-ci-samples/kaggle-ci-sample/visualize_table/Dockerfile (dockerfile)
  • samples/contrib/versioned-pipeline-ci-samples/mnist-ci-sample/tensorboard/Dockerfile (dockerfile)
  • samples/contrib/versioned-pipeline-ci-samples/mnist-ci-sample/train/Dockerfile (dockerfile)
  • test/api-integration-test/Dockerfile (dockerfile)
  • test/frontend-integration-test/Dockerfile (dockerfile)
  • test/frontend-integration-test/selenium-standalone-chrome-gcloud-nodejs.Docker/Dockerfile (dockerfile)
  • test/imagebuilder/Dockerfile (dockerfile)
  • test/images/Dockerfile (dockerfile)
  • test/initialization-test/Dockerfile (dockerfile)
  • test/sample-test/Dockerfile (dockerfile)
  • tools/bazel_builder/Dockerfile (dockerfile)
  • go.mod (gomod)
  • frontend/mock-backend/package.json (npm)
  • frontend/package.json (npm)
  • frontend/server/package.json (npm)
  • package.json (npm)
  • test/frontend-integration-test/package.json (npm)
  • frontend/.nvmrc (nvm)
  • backend/metadata_writer/requirements.txt (pip_requirements)
  • backend/requirements.txt (pip_requirements)
  • backend/src/apiserver/visualization/requirements.txt (pip_requirements)
  • components/kubeflow/katib-launcher/requirements.txt (pip_requirements)
  • contrib/components/openvino/ovms-deployer/containers/requirements.txt (pip_requirements)
  • docs/requirements.txt (pip_requirements)
  • samples/contrib/azure-samples/databricks-pipelines/requirements.txt (pip_requirements)
  • samples/contrib/ibm-samples/ffdl-seldon/source/seldon-pytorch-serving-image/requirements.txt (pip_requirements)
  • samples/core/ai_platform/training/requirements.txt (pip_requirements)
  • samples/core/container_build/requirements.txt (pip_requirements)
  • sdk/python/requirements.txt (pip_requirements)
  • test/kfp-functional-test/requirements.txt (pip_requirements)
  • test/sample-test/requirements.txt (pip_requirements)
  • components/gcp/container/component_sdk/python/setup.py (pip_setup)
  • components/kubeflow/dnntrainer/src/setup.py (pip_setup)
  • samples/core/ai_platform/training/setup.py (pip_setup)
  • sdk/python/setup.py (pip_setup)

Configuration

🔡 Renovate has detected a custom config for this PR. Feel free to ask for help if you have any doubts and would like it reviewed.

Important: Now that this branch is edited, Renovate can't rebase it from the base branch any more. If you make changes to the base branch that could impact this onboarding PR, please merge them manually.

What to Expect

With your current configuration, Renovate will create 57 Pull Requests:

chore(deps): pin dependencies
chore(deps): update gcr.io/inverting-proxy/agent docker digest to 9817c74
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/docker-gcr.io-inverting-proxy-agent
  • Merge into: master
  • Upgrade gcr.io/inverting-proxy/agent to sha256:9817c740a3705e4bf889e612c071686a8cb3cfcfe9ad191c570a295c37316ff0
chore(deps): update github.com/vividcortex/mysqlerr commit hash to 4c396ae
chore(deps): update golang.org/x/net commit hash to 5f4716e
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/golang.org-x-net-digest
  • Merge into: master
  • Upgrade golang.org/x/net to 5f4716e94777e714bc2fb3e3a44599cb40817aac
chore(deps): update google.golang.org/genproto commit hash to 646a494
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/google.golang.org-genproto-digest
  • Merge into: master
  • Upgrade google.golang.org/genproto to 646a494a81eaa116cb3e3978e5ac1278e35abfdd
chore(deps): update docker patch updates docker tags (patch)
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/docker-patch-docker-updates
  • Merge into: master
  • Upgrade golang to 1.13.15-stretch
  • Upgrade tensorflow/tensorflow to 2.0.4-py3
  • Upgrade tensorflow/tensorflow to 2.2.2
chore(deps): update go.mod dependencies (patch)
fix(deps): update npm dependencies (patch)
chore(deps): update alpine docker tag to v3.13
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/docker-alpine-3.x
  • Merge into: master
  • Upgrade alpine to 3.13
chore(deps): update gcr.io/cloud-marketplace-tools/k8s/deployer_helm/onbuild docker tag to v0.10.10
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/docker-gcr.io-cloud-marketplace-tools-k8s-deployer_helm-onbuild-0.x
  • Merge into: master
  • Upgrade gcr.io/cloud-marketplace-tools/k8s/deployer_helm/onbuild to 0.10.10
chore(deps): update go.mod dependencies (minor)
chore(deps): update golang docker tag
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/docker-golang-1.x
  • Merge into: master
  • Upgrade golang to 1.15.7
  • Upgrade golang to 1.15.7-alpine3.12
  • Upgrade golang to 1.14.14-stretch
chore(deps): update node.js
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/node-12.x
  • Merge into: master
  • Upgrade node to 12.20.1
  • Upgrade node to 12.20.1-alpine
chore(deps): update npm dependencies (minor)
chore(deps): update nvcr.io/nvidia/tensorflow docker tag to v19.10
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/docker-nvcr.io-nvidia-tensorflow-19.x
  • Merge into: master
  • Upgrade nvcr.io/nvidia/tensorflow to 19.10-py3
chore(deps): update python docker tag to v3.9
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/docker-python-3.x
  • Merge into: master
  • Upgrade python to 3.9-slim
  • Upgrade python to 3.9
chore(deps): update tensorflow/tensorflow docker tag
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/docker-tensorflow-tensorflow-2.x
  • Merge into: master
  • Upgrade tensorflow/tensorflow to 2.2.2-py3
  • Upgrade tensorflow/tensorflow to 2.4.1
chore(deps): update dependency @testing-library/react to v11
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/testing-library-react-11.x
  • Merge into: master
  • Upgrade @testing-library/react to 11.2.3
chore(deps): update dependency @​types/jest to v26
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/jest-26.x
  • Merge into: master
  • Upgrade @types/jest to 26.0.20
chore(deps): update dependency @​types/react to v17
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/react-17.x
  • Merge into: master
  • Upgrade @types/react to 17.0.0
chore(deps): update dependency @​types/react-dom to v17
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/react-dom-17.x
  • Merge into: master
  • Upgrade @types/react-dom to 17.0.0
chore(deps): update dependency @​types/react-router-dom to v5
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/react-router-dom-5.x
  • Merge into: master
  • Upgrade @types/react-router-dom to 5.1.7
chore(deps): update dependency @​types/react-test-renderer to v17
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/react-test-renderer-17.x
  • Merge into: master
  • Upgrade @types/react-test-renderer to 17.0.0
chore(deps): update dependency @​types/tar-stream to v2
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/tar-stream-2.x
  • Merge into: master
  • Upgrade @types/tar-stream to 2.2.0
chore(deps): update dependency jest to v26
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/major-jest-monorepo
  • Merge into: master
  • Upgrade jest to 26.6.3
chore(deps): update dependency prettier to v2
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/prettier-2.x
  • Merge into: master
  • Upgrade prettier to 2.2.1
  • Upgrade @types/prettier to 2.1.6
chore(deps): update dependency react-scripts to v4
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/react-scripts-4.x
  • Merge into: master
  • Upgrade react-scripts to 4.0.1
chore(deps): update dependency standard-version to v9
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/standard-version-9.x
  • Merge into: master
  • Upgrade standard-version to 9.1.0
chore(deps): update dependency supertest to v6
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/supertest-6.x
  • Merge into: master
  • Upgrade supertest to 6.1.3
chore(deps): update dependency ts-jest to v26
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/ts-jest-26.x
  • Merge into: master
  • Upgrade ts-jest to 26.5.0
chore(deps): update dependency ts-node to v9
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/ts-node-9.x
  • Merge into: master
  • Upgrade ts-node to 9.1.1
chore(deps): update dependency typescript to v4
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/typescript-4.x
  • Merge into: master
  • Upgrade typescript to 4.1.3
chore(deps): update dependency webpack to v5
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/webpack-5.x
  • Merge into: master
  • Upgrade webpack to 5.19.0
chore(deps): update dependency webpack-bundle-analyzer to v4
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/webpack-bundle-analyzer-4.x
  • Merge into: master
  • Upgrade webpack-bundle-analyzer to 4.4.0
chore(deps): update module argoproj/argo to v2
chore(deps): update module cenkalti/backoff to v4
chore(deps): update module grpc-ecosystem/grpc-gateway to v2
chore(deps): update module k8s.io/client-go to v12
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/k8s.io-client-go-12.x
  • Merge into: master
  • Upgrade k8s.io/client-go to v12.0.0
chore(deps): update module masterminds/squirrel to v1
chore(deps): update module mattn/go-sqlite3 to v2
chore(deps): update module minio/minio-go to v7
chore(deps): update module robfig/cron to v3
fix(deps): update dependency @google-cloud/storage to v5
fix(deps): update dependency crypto-js to v4
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/crypto-js-4.x
  • Merge into: master
  • Upgrade crypto-js to ^4.0.0
  • Upgrade @types/crypto-js to 4.0.1
fix(deps): update dependency d3 to v6
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/d3-6.x
  • Merge into: master
  • Upgrade d3 to 6.5.0
  • Upgrade @types/d3 to 6.3.0
fix(deps): update dependency d3-dsv to v2
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/d3-dsv-2.x
  • Merge into: master
  • Upgrade d3-dsv to 2.0.0
  • Upgrade @types/d3-dsv to 2.0.1
fix(deps): update dependency http-proxy-middleware to v1
fix(deps): update dependency js-yaml to v4
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/js-yaml-4.x
  • Merge into: master
  • Upgrade js-yaml to 4.0.0
  • Upgrade @types/js-yaml to 4.0.0
fix(deps): update dependency markdown-to-jsx to v7
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/markdown-to-jsx-7.x
  • Merge into: master
  • Upgrade markdown-to-jsx to 7.1.1
fix(deps): update dependency mocha to v8
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/mocha-8.x
  • Merge into: master
  • Upgrade mocha to 8.2.1
fix(deps): update dependency re-resizable to v6
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/re-resizable-6.x
  • Merge into: master
  • Upgrade re-resizable to 6.9.0
fix(deps): update dependency react-ace to v9
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/react-ace-9.x
  • Merge into: master
  • Upgrade react-ace to 9.3.0
fix(deps): update dependency react-dropzone to v11
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/react-dropzone-11.x
  • Merge into: master
  • Upgrade react-dropzone to 11.2.4
fix(deps): update dependency react-router-dom to v5
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/major-reactrouter-monorepo
  • Merge into: master
  • Upgrade react-router-dom to 5.2.0
fix(deps): update dependency webdriverio to v6
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/major-webdriverio-monorepo
  • Merge into: master
  • Upgrade webdriverio to 6.12.1
fix(deps): update mui monorepo (major)
fix(deps): update react monorepo to v17 (major)
  • Schedule: ["before 3am on Monday"]
  • Branch name: renovate/major-react-monorepo
  • Merge into: master
  • Upgrade react to 17.0.1
  • Upgrade react-dom to 17.0.1
  • Upgrade react-test-renderer to 17.0.1

🚸 Branch creation will be limited to maximum 2 per hour, so it doesn't swamp any CI resources or spam the project. See docs for prhourlylimit for details.


❓ Got questions? Check out Renovate's Docs, particularly the Getting Started section.
If you need any further assistance then you can also request help here.


This PR has been generated by WhiteSource Renovate. View repository job log here.

@google-oss-robot
Copy link

Hi @renovate-bot. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@rarkins
Copy link
Contributor

rarkins commented Jan 29, 2021

@davidspek I think you've already been experimenting with a config for this repo? Feel free to @ me to review it before merging

@davidspek
Copy link
Contributor

@rarkins @Bobgy This is what I had so far:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:base",
    "schedule:daily",
    ":dependencyDashboard",
    ":rebaseStalePrs",
    ":semanticCommits",
    ":semanticCommitScope(deps)",
    "docker:enableMajor",
    ":masterIssue",
    "group:linters",
    "group:polymer",
    "group:googleapis",
    "group:goOpenapi",
    "default:automergeLinters"
  ],
  "lockFileMaintenance": {
    "enabled": true,
    "automergeType": "branch",
    "dependencyDashboardApproval": true
  },
  "packageRules": [
    {
      "updateTypes": ["major"],
      "dependencyDashboardApproval": true
    },
    {
        "datasources": ["docker"],
        "updateTypes": ["patch", "minor"],
        "groupName": "docker patch & minor updates"
    },
    {
        "depTypeList": ["go_repository"],
        "updateTypes": ["patch", "minor"],
        "groupName": "Bazel workspace dependencies"

    }
  ],
  "ignorePaths": ["components/deprecated/"],
  "dependencyDashboard": true,
  "dependencyDashboardApproval": true,
  "separateMinorPatch": true
}

Notice, I specifically set "dependencyDashboardApproval": true in the lockFileMaintenance section for this initial add.

@davidspek
Copy link
Contributor

Grouping the docker updates should maybe be configured for patch updates only, as minor updates will need specific folder rules.

@rarkins
Copy link
Contributor

rarkins commented Jan 29, 2021

Anything automerge-related won't be possible (e.g. "automergeType": "branch",).

If you commit your renovate.json over the top of this branch, the onboarding PR description will be updated to reflect the new config. i.e. just add a new commit with what you have above

@Bobgy
Copy link
Contributor

Bobgy commented Jan 29, 2021

Some early comments I'd suggest adding experience maintaining this repo:

we should disable dockerfile upgrade for everything in our kustomize manifests and non-language third party images, because Google has strict license compliance requirements and we need to update their license files during an upgrade, that cannot be automated by renovate bot (therefore, is it possible to only turn on the dashboard feature for them?).

still looking at the documentation

@Bobgy
Copy link
Contributor

Bobgy commented Jan 29, 2021

I'll iterate on the config a few times to remove all files we do not want to auto update.

renovate.json Outdated Show resolved Hide resolved
renovate.json Outdated Show resolved Hide resolved
@Bobgy
Copy link
Contributor

Bobgy commented Jan 29, 2021

@renovate-bot rescan
(is this possible?)

@rarkins
Copy link
Contributor

rarkins commented Jan 29, 2021

It should, but each new commit will cause any existing run to abort until the next run. It takes quite a while to scan the repo

@chensun
Copy link
Member

chensun commented Jan 31, 2021

@rarkins @davidspek Thanks for introducing the tool.

I have some questions regarding the Python dependency updates.
Taking the following change as an example, which is from davidspek#163
image

  1. How does the bot decide which version to pin? Do you reuse pip's resolver under the hood? If so, do you always use the latest pip? and if not, won't there be a chance that your resolver decides something different than pip?
  2. For tensorflow>=1.13,<2, is the upper limit <2 stored anywhere? How would future updates know there was an upper restriction?

@rarkins
Copy link
Contributor

rarkins commented Jan 31, 2021

@chensun

  1. It uses pep440 versioning according to the Python spec and does not rely on pip directly.
  2. The upper limit is no longer required because once you pin the version, it's never going to install >=2.0.0 anyway. Major updates in Renovate are treated as special and there's no chance you'll "accidentally" get updated past 2.0.0. If for some reason this is considered as a "never ever upgrade past 2.0, we'll be using 1.x until the end of time" rule then you can add a package rule in Renovate with "allowedVersions": "<2" in which case Renovate will pretend like >=2 never exists and you'll never see a PR suggested or anything in the dashboard

@rarkins
Copy link
Contributor

rarkins commented Jan 31, 2021

BTW I can't comment specifically on your dependencies like numpy, pandas and six, but having constraints like >=0.22 is usually a very bad idea. Even when a new release is major and documented as breaking, pip will install it for you and you might find multiple things breaking. Never use uncapped constraints unless you have a very good reason for it.

@davidspek
Copy link
Contributor

While we are on the topic of python dependencies. I think it is worth the effort to go through all the Dockerfiles and remove any RUN pip install.. commands in favour for having the dependencies in a requirements.txt file alongside the Dockerfile which then gets copied into the container for installation. This way Renovate can keep all the python dependencies in the containers up to date as well.

@Bobgy
Copy link
Contributor

Bobgy commented Jan 31, 2021

@chensun

  1. It uses pep440 versioning according to the Python spec and does not rely on pip directly.
  2. The upper limit is no longer required because once you pin the version, it's never going to install >=2.0.0 anyway. Major updates in Renovate are treated as special and there's no chance you'll "accidentally" get updated past 2.0.0. If for some reason this is considered as a "never ever upgrade past 2.0, we'll be using 1.x until the end of time" rule then you can add a package rule in Renovate with "allowedVersions": "<2" in which case Renovate will pretend like >=2 never exists and you'll never see a PR suggested or anything in the dashboard

I can understand that, the best way to integrate with a tool is moving your source of truth to it... thus we will need to manage all versions in renovate package rules.

Is there a less intrusive way to integrate?
We adopted the workflow to define python package requirements in https://github.com/kubeflow/pipelines/blob/0795597562e076437a21745e524b5c960b1edb68/backend/src/apiserver/visualization/requirements.in, and compile to full requirements.txt using pip-compile.

The approach is similar to package.json and package-lock.json, is there a similar workflow we can use in python that is compatible with renovate?

EDIT: I found it, looks like renovate bot has native support for python-poetry: https://docs.renovatebot.com/modules/manager/poetry/. Poetry is a package manager like npm for python. Maybe we should consider poetry instead.

Note, all these discussion do not need to block the configuration PR, because python dependencies were disabled from auto-upgrade in the config.

@Bobgy
Copy link
Contributor

Bobgy commented Jan 31, 2021

@davidspek that's a good point! Would you mind creating an issue to track it?

@davidspek
Copy link
Contributor

@Bobgy I will create a PR for it once I have figured out why I can't get an image to start services with the s6 overlay :).

@davidspek
Copy link
Contributor

@Bobgy PR to remove the pip dependencies from Dockerfiles to a requirements.txt file: #5064. Will need some review to see if it makes sense with the CI building and such.

@chensun
Copy link
Member

chensun commented Feb 1, 2021

@chensun

  1. It uses pep440 versioning according to the Python spec and does not rely on pip directly.
  2. The upper limit is no longer required because once you pin the version, it's never going to install >=2.0.0 anyway. Major updates in Renovate are treated as special and there's no chance you'll "accidentally" get updated past 2.0.0. If for some reason this is considered as a "never ever upgrade past 2.0, we'll be using 1.x until the end of time" rule then you can add a package rule in Renovate with "allowedVersions": "<2" in which case Renovate will pretend like >=2 never exists and you'll never see a PR suggested or anything in the dashboard

@rarkins, thanks for explaining this. So if I understand correctly, the bot doesn't have a dependency resolver. All it does is to look at requirements.txt and update each entry with the latest minor release. It doesn't really care if the packages are compatible (theoretically minor version update shouldn't break compatibility). Am I right?

The reason I mentioned pip is because we recent hit some issue with the new resolver logic comes with pip 20.3.* (even though it's a minor update, it actually broke us). Context: #4853

pip is kinda special, but there might be cases we do want to choose a minor version as an upper constraint. For instance, during the early development phase, we may want to reserve 1.0.0 and possibly bump minor version when it should have been a major version (there can be violations of semantic versioning).

'kfp-pipeline-spec>=0.1.0, <0.2.0',

BTW I can't comment specifically on your dependencies like numpy, pandas and six, but having constraints like >=0.22 is usually a very bad idea. Even when a new release is major and documented as breaking, pip will install it for you and you might find multiple things breaking. Never use uncapped constraints unless you have a very good reason for it.

In general, I agree. I thought that requirements.txt is for development environment only, so it might be okay to pilot with newer versions even they are major updates? Can't speak for others, in my workflow, I don't actually use requirements.txt for kfp sdk development.

@Bobgy
Copy link
Contributor

Bobgy commented Feb 1, 2021

/ok-to-test

@chensun
Copy link
Member

chensun commented Feb 1, 2021

@chensun

  1. It uses pep440 versioning according to the Python spec and does not rely on pip directly.
  2. The upper limit is no longer required because once you pin the version, it's never going to install >=2.0.0 anyway. Major updates in Renovate are treated as special and there's no chance you'll "accidentally" get updated past 2.0.0. If for some reason this is considered as a "never ever upgrade past 2.0, we'll be using 1.x until the end of time" rule then you can add a package rule in Renovate with "allowedVersions": "<2" in which case Renovate will pretend like >=2 never exists and you'll never see a PR suggested or anything in the dashboard

I can understand that, the best way to integrate with a tool is moving your source of truth to it... thus we will need to manage all versions in renovate package rules.

For Python, the source of truth of dependency should probably be the setup.py file, in a sense that this is what being used when an end user install a package via pip install. requirements.txt on the other hand is to prepare a development enviroment IIUC.

Is there a less intrusive way to integrate?
We adopted the workflow to define python package requirements in https://github.com/kubeflow/pipelines/blob/0795597562e076437a21745e524b5c960b1edb68/backend/src/apiserver/visualization/requirements.in, and compile to full requirements.txt using pip-compile.

While I don't use the requirements.txt in my workflow, but I see one pros of this pip-compile tool is that it explains clearly how each package is determined. Once we have this renovate updates, that information might not be accurate then, we probably have to remove those comments and requirements.in

The approach is similar to package.json and package-lock.json, is there a similar workflow we can use in python that is compatible with renovate?

EDIT: I found it, looks like renovate bot has native support for python-poetry: https://docs.renovatebot.com/modules/manager/poetry/. Poetry is a package manager like npm for python. Maybe we should consider poetry instead.

For alternatives, I recently started using pipenv in a personal project. It's probably closer to "package.json and package-lock.json" as you mentioned with "Pipfile and Pipfile.lock". It also supports virtual env which is nice. That said, I'm still new to this tool, this comment is not a recommendation of switching to it in KFP.

Note, all these discussion do not need to block the configuration PR, because python dependencies were disabled from auto-upgrade in the config.

That's fine. We can experiment with this tool and see how it works.

@davidspek
Copy link
Contributor

/lgtm

@Bobgy
Copy link
Contributor

Bobgy commented Feb 2, 2021

Thanks, we can continue the python dependency discussion in related issues.
I'm merging this PR to first verify Renovate for other purpooses.
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bobgy, renovate-bot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bobgy, renovate-bot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@forking-renovate
Copy link

Renovate is disabled

Renovate is disabled due to lack of config. If you wish to reenable it, you can either (a) commit a config file to your base branch, or (b) rename this closed PR to trigger a replacement onboarding PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants