Automatic workspace discovery

[mrnugget]

What?

This adds automatic workspace discovery to src-cli. It allows users to

1. define where projects in large repositories live and
2. run the campaign specs steps in those project folders, turning them into workspaces.

How?

Users define workspaces like so:

workspaces:
  - rootAtLocationOf: go.mod
    in: github.com/sourcegraph/sourc*

That means: in every repository that starts with github.com/sourcegraph/sourc projects have a go.mod at its root and those folders should be used as workspaces for the execution of campaign spec steps.

src-cli uses Sourcegraph search under the hood to search for the locations of the rootAtLocationOf file, which means it doesn't need to download the repository first and search the file system.

workspaces can also contain multiple definitions, matching different repositories (but a repository cannot be matched by multiple definitions):

on:
  - repositoriesMatchingQuery: repo:automation-testing$
  - repository: github.com/sourcegraph/sourcegraph

workspaces:
  - rootAtLocationOf: go.mod
    in: github.com/sourcegraph/sourcegraph*
  - rootAtLocationOf: package.json
    in: "*automation-testing"

Since multiple workspaces per repository means that multiple changesets will be produced in a single repository, the changesetTemplate.branch needs to use templating to avoid name clashes.

For that, users can access the template variable steps.path and use helper functions to generate a unique branch name per changeset. Example:

changesetTemplate:
  # [...]
  branch: ${{ join_if "-" "thorsten/workspace-discovery" (replace steps.path "/" "-") }}
  published: false
  commit:
    message: Automatic workspace discovery

(The join_if and the replace helpers are new. join_if joins the given list of strings, but ignoring the blank strings. replace is an alias for strings.ReplaceAll)

Users can, of course, also user other ways to generate a unique branch name per changeset. With outputs, for example:

steps:
  # [...]
  - run:  if [[ -f "package.json" ]]; then cat package.json | jq -j .name; fi
    container: jiapantw/jq-alpine:latest
    outputs:
      projectName:
        value: ${{ step.stdout }}

changesetTemplate:
  # [...]
  branch: thorsten/workspace-discovery-${{ outputs.projectName }}

Or, in combination:

steps:
  # [...]
  - run:  if [[ -f "package.json" ]]; then cat package.json | jq -j .name; fi
    container: jiapantw/jq-alpine:latest
    outputs:
      projectName:
        value: ${{ step.stdout }}

changesetTemplate:
  # If we have an `outputs.projectName` we use it, otherwise we append the path
  # of the workspace. If the path is emtpy (as is the case in the root folder),
  # we ignore it.
  branch: |
    ${{ if eq outputs.projectName "" }}
    ${{ join_if "-" "thorsten/workspace-discovery" (replace steps.path "/" "-") }}
    ${{ else }}
    thorsten/workspace-discovery-${{ outputs.projectName }}
    ${{ end }}

Details & Edge Cases

• Repositories matching multiple workspace definitions produce an error.
• If a repository is yielded by on and not matched by an workspaces.in: glob, the steps will be executed in its root folder.
• If a repository is yielded by on and matches a workspace.in: glob, but there are no workspaces in it that contain the file in rootAtLocationOf then the steps won't be executed in the repository.
• If a repository contains multiple workspaces where one is the subdirectory of another, then the steps will be executed in both and the parent directory also includes the sub directory. I think this is the most intuitive choice.

Dependency

This requires the addition of workspaces to the campaign spec schema, which means it requires changes to the Sourcegraph server.

The PR is here: https://github.com/sourcegraph/sourcegraph/pull/17757

What's not included

src-cli still downloads a complete archive of every matched repository, even if the steps should only be included in subdirectories. Only downloading archives of the workspace directories is something we should implement in the near future to make support for large monorepos better.

Full campaign spec to try this at home

There you go:

name: workspace-discovery
description: Automatic workspace discovery

on:
  - repositoriesMatchingQuery: repo:automation-testing$
  - repository: github.com/sourcegraph/sourcegraph

workspaces:
  - rootAtLocationOf: go.mod
    in: github.com/sourcegraph/sourcegraph*
  - rootAtLocationOf: package.json
    in: "*automation-testing"

steps:
  - run: "echo \"pwd: $(pwd)\nfiles: $(ls)\" >> message.txt"
    container: alpine:3
  - run:  if [[ -f "package.json" ]]; then cat package.json | jq -j .name; fi
    container: jiapantw/jq-alpine:latest
    outputs:
      projectName:
        value: ${{ step.stdout }}

changesetTemplate:
  title: Automatic workspace discovery
  body:  Automatic workspace discovery

  # If we have an `outputs.projectName` we use it, otherwise we append the path
  # of the workspace. If the path is emtpy (as is the case in the root folder),
  # we ignore it.
  branch: |
    ${{ if eq outputs.projectName "" }}
    ${{ join_if "-" "thorsten/workspace-discovery" (replace steps.path "/" "-") }}
    ${{ else }}
    thorsten/workspace-discovery-${{ outputs.projectName }}
    ${{ end }}
  published: false
  commit:
    message: Automatic workspace discovery

[mrnugget]

Heads up reviewers: just noticed that I forgot to add a validation that produces an error and human readable message when multiple changesets with the same branch have been produced in the same repository. I'll add that, but that shouldn't stop anyone from reviewing this.

[eseliger]

Looks good to me 🌟

[eseliger]

Wouldn't this mean when using 1 worker thread, there would be no caching at all? and can there be the case where the workers are utilized like the following

[repo A:/path1]
[repo B]
[repo C]
[repo D]

=>

[repo A:/path2]
[repo B]
[repo C]
[repo D]

where the worker would close the zip for repo A at path1, and then need to refetch for path2?

[mrnugget]

Nice catch. You're right. I'll address in follow-up PR.

[mrnugget]

There are two things that I need to do:

• Validate and throw an error if multiple changesets have the same branch. See: Check for branch duplicates after creating changeset specs #448
• Make sure that repository archives are not downloaded multiple times if tasks in the same repository run sequentially and not concurrently. See: Keep repo archives cached for later executed tasks #449

I'll address those in follow-up PRs to make it easier for review.