Skip to content

Renovate integration

Jump to bottom
Stéphane Brunner edited this page May 31, 2024 · 4 revisions

Common usage of Renovate

Introduction

Personally, I always use exact pinning of all the dependency to have better known leg of the updates, If you don't do that, you have many things in the package lock pull request, without any details in the pull request description.

To publish a package with non-exact version in the dependency, see Poetry section.

Integration

Get the workflow pull-request-automation.yaml to review automatically the CVE, dpkg and Renovate updates.

Base

  extends: ['config:base'],  
  timezone: 'Europe/Zurich',
  schedule: 'after 5pm on the first day of the month',
  labels: ['dependencies'],
  separateMajorMinor: true,
  separateMinorPatch: true,
  prHourlyLimit: 0,
  prConcurrentLimit: 0,
  lockFileMaintenance: {
    enabled: true,
    automerge: true,
    schedule: 'after 5pm on the first day of the month',
  },

Run on stabilization branches

  baseBranches: ['x.y', ..., 'master'],
  packageRules: [
    /** Accept only the patch on stabilization branches */
    {
      matchBaseBranches: ['/^[0-9]+\\.[0-9]+$/', '/release_.*/'],
      matchUpdateTypes: ['major', 'minor', 'pin', 'digest', 'lockFileMaintenance', 'rollback', 'bump'],
      enabled: false,
    },
  ]

Pre-commit

In the .pre-commit-config.yaml file add # npm or # pip after all the additional_dependencies.

Use the following Renovate configuration:

  'pre-commit': { enabled: true },
  regexManagers: [
    /** Do updates on pre-commit additional dependencies */
    {
      fileMatch: ['^\\.pre\\-commit\\-config\\.yaml$'],
      matchStrings: [" +- '?(?<depName>[^' @=]+)(@|==)(?<currentValue>[^' @=]+)'? # (?<datasource>.+)"],
    },
  ],

C2cciutils

To update the used schema version in the config.yaml, like:

# yaml-language-server: $schema=https://raw.githubusercontent.com/camptocamp/c2cciutils/1.6.15/c2cciutils/schema.json

use the following configuration:

  regexManagers: [
    /** Do update on the schema present in the ci/config.yaml */
    {
      fileMatch: ['^ci/config\\.yaml$'],
      matchStrings: [
        '.*https://raw\\.githubusercontent\\.com/(?<depName>[^\\s]+)/(?<currentValue>[0-9\\.]+)/.*',
      ],
      datasourceTemplate: 'github-tags',
    },
  ],

By the way, this line is used by VSCode to add some documentation and validation while you edit the configuration file.

Common package rules

  packageRules: [
    /** Auto merge the dev dependency update */
    {
      matchDepTypes: ['devDependencies'],
      automerge: true,
    },
    /** Group and auto merge the patch updates */
    {
      matchUpdateTypes: ['patch'],
      groupName: 'all patch versions',
      automerge: true,
    },
    /** Group and auto merge the minor updates */
    {
      matchUpdateTypes: ['minor'],
      groupName: 'all minor versions',
      automerge: true,
    },
    /** Group Poetry packages */
    {
      matchPackageNames: ['poetry', 'pip'],
      matchPackagePrefixes: ['poetry-'],
      groupName: 'Poetry',
      automerge: true,
    },
    /** Group and auto merge the CI dependencies */
    {
      matchFileNames: ['.github/**', '.pre-commit-config.yaml', 'ci/**'],
      groupName: 'CI dependencies',
      automerge: true,
    },
  ]

Poetry integration

I want to pin all the dependency, then with this requirement the requirement.txt is not a good choice...

I try to use Pipenv, but I get many issue during the lock (and also Renovate).

Finally, I use Poetry that I found that a really good choice, and I like the --no-update option to apply a security fix on a stabilization branch :-).

I use it with the following plugin's configuration to have a good integration:

[tool.poetry]
version = "0.0.0"

[build-system]
requires = [
  "poetry-core>=1.0.0",
  "poetry-dynamic-versioning[plugin]",
  "poetry-plugin-tweak-dependencies-version",
  "poetry-plugin-drop-python-upper-constraint",
]
build-backend = "poetry.core.masonry.api"

[tool.poetry-dynamic-versioning]
enable = true
vcs = "git"
pattern = "^(?P<base>\\d+(\\.\\d+)*)"
format-jinja = """
{%- if env.get("VERSION_TYPE") == "version_branch" -%}
{{serialize_pep440(bump_version(base, 1 if env.get("IS_MASTER") == "TRUE" else 2), dev=distance)}}
{%- elif distance == 0 -%}
{{serialize_pep440(base)}}
{%- else -%}
{{serialize_pep440(bump_version(base), dev=distance)}}
{%- endif -%}
"""

[tool.poetry-plugin-tweak-dependencies-version]
default = "present"

poetry-dynamic-versioning is used to by able to publish the package with a tag without having to update the version in the package description. This works because c2cciutils set the following environment variable to create the published package:

  • VERSION: the published version type.
  • VERSION_TYPE: the version type version_tag, version_branch, feature_branch, feature_tag (for pull request).
  • IS_MASTER: if we are on the default branch (TRUE or FALSE).

poetry-plugin-tweak-dependencies-version is used to be able to do not pin exact version of the dependency, with this configuration we just require the dependency, but the plugin is really flexible, see plugin documentation for more details.

poetry-plugin-drop-python-upper-constraint is used because to be able to do the lock poetry requires an upper python version, but usually, we don't want to limit it in the published version.

Dockerfile

Common integration in the Dockerfile.

FROM ubuntu:22.04 AS base-all
LABEL maintainer Camptocamp "[email protected]"
SHELL ["/bin/bash", "-o", "pipefail", "-cux"]

RUN --mount=type=cache,target=/var/lib/apt/lists --mount=type=cache,target=/var/cache \
    sed -i '/-backports /d' /etc/apt/sources.list \
    && apt-get update \
    && apt-get install --yes --no-install-recommends python3-pip binutils

# Used to convert the locked packages by poetry to pip requirements format
# We don't directly use `poetry install` because it force to use a virtual environment.
FROM base-all as poetry

# Install Poetry
WORKDIR /tmp
COPY requirements.txt ./
RUN --mount=type=cache,target=/root/.cache \
    python3 -m pip install --disable-pip-version-check --requirement=requirements.txt

# Do the conversion
COPY poetry.lock pyproject.toml ./
ENV POETRY_DYNAMIC_VERSIONING_BYPASS=0.0.0
RUN poetry export --extras=checks --extras=publish --extras=audit --extras=version --output=requirements.txt \
    && poetry export --with=dev --output=requirements-dev.txt

# Base, the biggest thing is to install the Python packages
FROM base-all as base

RUN --mount=type=cache,target=/root/.cache \
    --mount=type=bind,from=poetry,source=/tmp,target=/poetry \
    python3 -m pip install --disable-pip-version-check --no-deps --requirement=/poetry/requirements.txt

FROM base AS dev

RUN --mount=type=cache,target=/root/.cache \
    --mount=type=bind,from=poetry,source=/tmp,target=/poetry \
    python3 -m pip install --disable-pip-version-check --no-deps --requirement=/poetry/requirements-dev.txt

FROM base AS run
    ...

With the requirements.txt with:

poetry==1.2.1
poetry-plugin-export==1.1.0
poetry-dynamic-versioning[plugin]==0.19.0
poetry-plugin-tweak-dependencies-version==1.0.0
Clone this wiki locally