Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(manager): Cloud Native Buildpacks project descriptor manager #30799

Merged
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
11085f9
feat: Cloud Native Buildpacks project descriptor manager
pbusko Aug 14, 2024
b4b0308
remove buildpacks category
pbusko Aug 15, 2024
5eedad0
rename manager to "buildpacks"
pbusko Aug 15, 2024
6606a44
use zod for schema validation
pbusko Aug 15, 2024
2eddca1
log parsing error
pbusko Aug 16, 2024
48f62cf
improve schema validation for buildpacks
pbusko Aug 16, 2024
dd4f612
rename manager to cnb
pbusko Aug 16, 2024
e42ed6d
Update lib/modules/manager/cnb/readme.md
pbusko Aug 16, 2024
89cbc3a
Update lib/modules/manager/cnb/readme.md
pbusko Aug 16, 2024
0f811a0
Apply suggestions from code review
pbusko Aug 16, 2024
3c8b794
typo fix
pbusko Aug 16, 2024
ab9c4d4
apply feedback suggestions
pbusko Aug 19, 2024
17944ad
make top-level io object optional
pbusko Aug 19, 2024
04f2d45
add packageFile to the debug log
pbusko Aug 19, 2024
bca2e5b
ignore unsupported buildpack sources
pbusko Aug 21, 2024
3c42ae3
remove unnecessary char escape
pbusko Aug 21, 2024
cb30642
add more validation for the buildpack uris
pbusko Aug 21, 2024
b5b6459
rename manager to "buildpacks"
pbusko Aug 23, 2024
b0060a8
simplify commit message topic
pbusko Aug 24, 2024
8d9c233
Invert detection logic
pbusko Oct 14, 2024
6a974c5
Adapt fixtures
pbusko Oct 14, 2024
df16d10
Simplify regex
pbusko Oct 14, 2024
d34a38b
optimise docker reference regex
pbusko Nov 12, 2024
4598172
Merge branch 'main' into feat/cnb-project-descriptor-support
pbusko Nov 13, 2024
2a70caf
allow longer sha256 digest strings
pbusko Nov 13, 2024
1b8eb48
do not allow . or - as starting symbol
pbusko Nov 13, 2024
40f7dc7
make regex for the registry hostname more strict
pbusko Nov 13, 2024
e8d9093
fix typo
pbusko Nov 13, 2024
1229679
reduce the number of groupings
pbusko Nov 13, 2024
c7cd604
Merge branch 'main' into feat/cnb-project-descriptor-support
loewenstein-sap Nov 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions lib/config/options/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1135,6 +1135,7 @@ const options: RenovateOptions[] = [
supportedManagers: [
'ansible',
'bitbucket-pipelines',
'buildpacks',
'crossplane',
'devcontainer',
'docker-compose',
Expand Down
2 changes: 2 additions & 0 deletions lib/modules/manager/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import * as bicep from './bicep';
import * as bitbucketPipelines from './bitbucket-pipelines';
import * as bitrise from './bitrise';
import * as buildkite from './buildkite';
import * as buildpacks from './buildpacks';
import * as bun from './bun';
import * as bunVersion from './bun-version';
import * as bundler from './bundler';
Expand Down Expand Up @@ -117,6 +118,7 @@ api.set('bicep', bicep);
api.set('bitbucket-pipelines', bitbucketPipelines);
api.set('bitrise', bitrise);
api.set('buildkite', buildkite);
api.set('buildpacks', buildpacks);
api.set('bun', bun);
api.set('bun-version', bunVersion);
api.set('bundler', bundler);
Expand Down
111 changes: 111 additions & 0 deletions lib/modules/manager/buildpacks/extract.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
import { codeBlock } from 'common-tags';

import { extractPackageFile } from '.';

describe('modules/manager/buildpacks/extract', () => {
describe('extractPackageFile()', () => {
it('returns null for invalid files', () => {
expect(extractPackageFile('not a project toml', '', {})).toBeNull();
});

it('returns null for empty package.toml', () => {
const res = extractPackageFile(
'[_]\nschema-version = "0.2"',
'project.toml',
{},
);
expect(res).toBeNull();
});

it('extracts builder and buildpack images', () => {
const res = extractPackageFile(
codeBlock`
[_]
schema-version = "0.2"

[io.buildpacks]
builder = "registry.corp/builder/noble:1.1.1"

[[io.buildpacks.group]]
uri = "docker://buildpacks/java:2.2.2"

[[io.buildpacks.group]]
uri = "buildpacks/nodejs:3.3.3"

[[io.buildpacks.group]]
uri = "example/[email protected]"

[[io.buildpacks.group]]
uri = "example/registry-cnb"

[[io.buildpacks.group]]
uri = "urn:cnb:registry:example/[email protected]"

[[io.buildpacks.group]]
uri = "some-bp@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"

[[io.buildpacks.group]]
uri = "cnbs/some-bp:some-tag@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"

[[io.buildpacks.group]]
uri = "from=builder:foobar"

[[io.buildpacks.group]]
uri = "file://local.oci"

[[io.buildpacks.group]]
uri = "foo://fake.oci"`,
'project.toml',
{},
);
expect(res?.deps).toEqual([

Check failure on line 61 in lib/modules/manager/buildpacks/extract.spec.ts

View workflow job for this annotation

GitHub Actions / test (3/16)

modules/manager/buildpacks/extract › extractPackageFile() › extracts builder and buildpack images

expect(received).toEqual(expected) // deep equality - Expected - 15 + Received + 3 Array [ Object { "autoReplaceStringTemplate": "{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}", "commitMessageTopic": "builder {{depName}}", + "currentDigest": undefined, "currentValue": "1.1.1", "datasource": "docker", "depName": "registry.corp/builder/noble", "replaceString": "registry.corp/builder/noble:1.1.1", }, Object { "autoReplaceStringTemplate": "{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}", + "currentDigest": undefined, "currentValue": "2.2.2", "datasource": "docker", "depName": "buildpacks/java", "replaceString": "buildpacks/java:2.2.2", }, Object { "autoReplaceStringTemplate": "{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}", + "currentDigest": undefined, "currentValue": "3.3.3", "datasource": "docker", "depName": "buildpacks/nodejs", "replaceString": "buildpacks/nodejs:3.3.3", - }, - Object { - "autoReplaceStringTemplate": "{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}", - "currentDigest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", - "datasource": "docker", - "depName": "some-bp", - "replaceString": "some-bp@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", - }, - Object { - "autoReplaceStringTemplate": "{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}", - "currentDigest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", - "currentValue": "some-tag", - "datasource": "docker", - "depName": "cnbs/some-bp", - "replaceString": "cnbs/some-bp:some-tag@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", }, ] at Object.<anonymous> (lib/modules/manager/buildpacks/extract.spec.ts:61:25)
{
autoReplaceStringTemplate:
'{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}',
commitMessageTopic: 'builder {{depName}}',
currentValue: '1.1.1',
datasource: 'docker',
depName: 'registry.corp/builder/noble',
replaceString: 'registry.corp/builder/noble:1.1.1',
},
{
autoReplaceStringTemplate:
'{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}',
currentValue: '2.2.2',
datasource: 'docker',
depName: 'buildpacks/java',
replaceString: 'buildpacks/java:2.2.2',
},
{
autoReplaceStringTemplate:
'{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}',
currentValue: '3.3.3',
datasource: 'docker',
depName: 'buildpacks/nodejs',
replaceString: 'buildpacks/nodejs:3.3.3',
},
{
autoReplaceStringTemplate:
'{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}',
currentDigest:
'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef',
datasource: 'docker',
depName: 'some-bp',
replaceString:
'some-bp@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef',
},
{
autoReplaceStringTemplate:
'{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}',
currentDigest:
'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef',
currentValue: 'some-tag',
datasource: 'docker',
depName: 'cnbs/some-bp',
replaceString:
'cnbs/some-bp:some-tag@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef',
},
]);
});
});
});
103 changes: 103 additions & 0 deletions lib/modules/manager/buildpacks/extract.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
import is from '@sindresorhus/is';
import { logger } from '../../../logger';
import { regEx } from '../../../util/regex';
import { getDep } from '../dockerfile/extract';
import type {
ExtractConfig,
PackageDependency,
PackageFileContent,
} from '../types';
import { type ProjectDescriptor, ProjectDescriptorToml } from './schema';

const dockerPrefix = regEx(/^docker:\/?\//);
const dockerRef = regEx(
/^((?:[\w.-]+(?:\.[\w.-]+)*)(?::\d+)?\/)?[a-z0-9]+((\.|_|__|-+)[a-z0-9]+)*(\/[a-z0-9]+((\.|_|__|-+)[a-z0-9]+)*)*(?::(\w[\w.-]{0,127})(?:@sha256:[A-Fa-f\d]{32})?|@sha256:[A-Fa-f\d]{32})$/,
Fixed Show fixed Hide fixed
);

function isDockerRef(ref: string): boolean {
if (ref.startsWith('docker:/') || dockerRef.test(ref)) {
rarkins marked this conversation as resolved.
Show resolved Hide resolved
return true;
}
return false;
}

function parseProjectToml(
content: string,
packageFile: string,
): ProjectDescriptor | null {
const res = ProjectDescriptorToml.safeParse(content);
if (res.success) {
return res.data;
}

logger.debug(
{ packageFile, err: res.error },
'Failed to parse buildpacks project descriptor TOML',
);

return null;
pbusko marked this conversation as resolved.
Show resolved Hide resolved
}

export function extractPackageFile(
content: string,
packageFile: string,
config: ExtractConfig,
): PackageFileContent | null {
const deps: PackageDependency[] = [];

const descriptor = parseProjectToml(content, packageFile);
if (!descriptor) {
return null;
}

if (
descriptor.io?.buildpacks?.builder &&
isDockerRef(descriptor.io.buildpacks.builder)
) {
const dep = getDep(
descriptor.io.buildpacks.builder.replace(dockerPrefix, ''),
true,
config.registryAliases,
);
logger.trace(
{
depName: dep.depName,
currentValue: dep.currentValue,
currentDigest: dep.currentDigest,
},
'Cloud Native Buildpacks builder',
);

deps.push({ ...dep, commitMessageTopic: 'builder {{depName}}' });
}

if (
descriptor.io?.buildpacks?.group &&
is.array(descriptor.io.buildpacks.group)
) {
for (const group of descriptor.io.buildpacks.group) {
if (group.uri && isDockerRef(group.uri)) {
const dep = getDep(
group.uri.replace(dockerPrefix, ''),
true,
config.registryAliases,
);
logger.trace(
{
depName: dep.depName,
currentValue: dep.currentValue,
currentDigest: dep.currentDigest,
},
'Cloud Native Buildpack',
);

deps.push(dep);
}
}
}

if (!deps.length) {
return null;
}
return { deps };
}
12 changes: 12 additions & 0 deletions lib/modules/manager/buildpacks/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import type { Category } from '../../../constants';
import { DockerDatasource } from '../../datasource/docker';
export { extractPackageFile } from './extract';

export const defaultConfig = {
commitMessageTopic: 'buildpack {{depName}}',
fileMatch: ['(^|/)project\\.toml$'],
pbusko marked this conversation as resolved.
Show resolved Hide resolved
pinDigests: false,
};

export const categories: Category[] = ['docker'];
export const supportedDatasources = [DockerDatasource.id];
26 changes: 26 additions & 0 deletions lib/modules/manager/buildpacks/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
The `buildpacks` manager updates Cloud Native Buildpacks project descriptors in `project.toml` files.
A `project.toml` file can reference builder / buildpack images by URIs.
Renovate can update a `project.toml` file if:

- It can find the file
- The file follows the [project descriptor specifications](https://github.com/buildpacks/spec/blob/main/extensions/project-descriptor.md)
- The buildpack `uri` is an OCI image reference (references to a local file or buildpack registry are ignored)

If you use buildpacks in the `io.buildpacks.group` array, then you _must_ configure the Docker reference (`uri`) for Renovate to work.

```toml title="Example of a project.toml file with Docker reference URIs"
[_]
schema-version = "0.2"

[io.buildpacks]
builder = "registry.corp/builder/noble:1.1.1"

[[io.buildpacks.group]]
uri = "docker://buildpacks/java:2.2.2"

[[io.buildpacks.group]]
uri = "buildpacks/nodejs:3.3.3"

[[io.buildpacks.group]]
uri = "file://local.oci" # will be ignored
```
25 changes: 25 additions & 0 deletions lib/modules/manager/buildpacks/schema.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import { z } from 'zod';
import { Toml } from '../../../util/schema-utils';

const BuildpackGroup = z.object({
uri: z.string().optional(),
});

const IoBuildpacks = z.object({
builder: z.string().optional(),
group: z.array(BuildpackGroup).optional(),
});

export const ProjectDescriptor = z.object({
_: z.object({
'schema-version': z.string(),
}),
io: z
.object({
buildpacks: IoBuildpacks.optional(),
})
.optional(),
});

export type ProjectDescriptor = z.infer<typeof ProjectDescriptor>;
export const ProjectDescriptorToml = Toml.pipe(ProjectDescriptor);
Loading