Azure Data Factory CI starter workflow

[fedeoliv]

This PR introduces the Azure Data Factory CI starter workflow.

[NinadKavimandan]

@fedeoliv LGTM. On the bubbling up front, we use repo analysis in the backend of GitHub now, and templates are recommended based on these tech_stacks.

These are the tech_stacks we currently detect - https://github.com/github-starter-workflows/repo-analysis-partner/blob/main/tech_stacks.yml

The detection engine is based on Linguist detection, so the criteria of detection are inline with languages.yml.
You can raise an issue to request detection of Azure Data Factory as a tech_stack.

[NinadKavimandan]

{
    "scripts":{
        "build":"node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
    },
    "dependencies":{
        "@microsoft/azure-data-factory-utilities":"^0.1.5"
    }
}

@fedeoliv This can be presented as a regex and that can be the detection criteria for Azure Data Factory.

[fedeoliv]

@NinadKavimandan thanks for the suggestion. I opened an issue to support ADF: Azure Data Factory support as a new tech_stack

[NinadKavimandan]

@fedeoliv is there any working example of this workflow?

[fedeoliv]

@NinadKavimandan I have one example here: https://github.com/fedeoliv/data-factory-workflows-demo.
I've been working on a step-by-step CI/CD tutorial for Azure docs (hopefully adding this starter template on the doc soon 😉)

[fedeoliv]

This is the Azure doc I'm working on: https://github.com/fedeoliv/azure-docs-pr/blob/data-factory-ci-cd-github/articles/data-factory/continuous-integration-delivery-github-actions.md

[NinadKavimandan]

@fedeoliv Thanks for pointing to the example!
Is having package.json with the aforementioned script not a requirement for the workflow to run successfully? I didn't see a package.json in your example repo either.

[fedeoliv]

@NinadKavimandan I made some refactoring on new versions of data-factory-validate and data-factory-export to simplify even more the GH actions usage. The package.json is not needed anymore as it's now embedded in a Dockerfile.

[NinadKavimandan]

@fedeoliv in that case, will package.json play any part?

[fedeoliv]

@NinadKavimandan not anymore. I was thinking the best option to identify ADF resources would be checking if the repo has at least one of the following folders: dataflow, dataset, integrationRuntime, linkedService, trigger or pipeline. If someone is going to create a brand new ADF with a GitHub repo without any custom files (e.g. publish_config.json), the minimal assets we'll have is at least one of these folders.

Is it possible to make these validations on the repo analysis?

[NinadKavimandan]

@fedeoliv detection criteria cannot be based on the directory name. Is the presence of publish_config.json good enough criteria?

[fedeoliv]

@NinadKavimandan that's a good one for sure :)

[NinadKavimandan]

@fedeoliv cool then! We can take either of these two routes now -

• Merge this PR now, I will add support for Azure Data Factory tech_stack and update categories for your template props

or

• You can update the categories to include Azure Data Factory right now and merge, whenever Azure Data Factory tech_stack is added to repo analysis, this template will be bubbled up.

What would you prefer? Either seems fine to me.

[NinadKavimandan]

Also
If there's any repo with publish_config.json handy, please share.

[fedeoliv]

@NinadKavimandan I updated the .properties to include the Azure Data Factory tech_stack. Also, I added a publish_config.json sample on my data-factory-workflows-demo repo. Thank you! 😉

[NinadKavimandan]

@fedeoliv thanks for pointing to the example. This is really helpful!
Will keep you posted about the developments on the repo analysis side.

Going ahead with the merge.

[NinadKavimandan]

Looks like I don't have the option to merge 😢
@ashwinsangem can you take a look and merge this PR?