Add Support for Azure Synapse

[mrpaulandrew]

Following initial investigations:
https://mrpaulandrew.com/2020/06/03/adf-procfwk-and-azure-synapse-orchestrate-preview-and-limitations/

Add support for Azure Synapse.

Include support for Worker and Framework pipelines in either Data Factory or Synapse.

[mrpaulandrew]

Severity Code Description Project File Line Suppression State
Error CS1061

'SynapseManagementClient' does not contain a definition for 'PipelineRuns' and no accessible extension method 'PipelineRuns' accepting a first argument of type 'SynapseManagementClient' could be found (are you missing a using directive or an assembly reference?)

Functions ...\GitHub\ADF.procfwk\Functions\CheckPipelineStatus.cs 110 Active

[mrpaulandrew]

To Do:

• Finish Execute Pipeline function once unblocked.
• Update Get Error Details function to include orchestrator name and type.
• Update Check Status function to include orchestrator name and type.
• Rename 'DataFactorys' table to 'Orchestrators' and replace through metadata database.
• Add orchestrator type attribute with check constraint.
• CREATE SYNONYM [procfwk].[DataFactorys] FOR [procfwk].[Orchestrators];

[mrpaulandrew]

Synapse still not ready.
Need pipeline feature parity between Data Factory and Synapse Orchestrate.

[mrpaulandrew]

@NowinskiK now Synapse has Git support, do you want to try deploying the framework pipelines there?

[NowinskiK]

I saw that yesterday. I didn't try but I believe it will not work.
The reason is simple: the definition of resources is different (i.e. resource path).
I will create the clone of my ADFTools and change the code accordingly and I will let you know.
Then we could back to the discussion we started recently via email.

[mrpaulandrew]

Thanks for the Function app refactoring @richardswinbank . Using these new foundations I've just created the bare bones Synapse service.

[mrpaulandrew]

Development of Azure Functions App now complete using the Synapse namespace:

Azure.Analytics.Synapse.Artifacts

With version 1.0.0-beta.1 of the NuGet package.