Working with Azure Data Factory Pipelines
Posted onI am often tasked with automating tasks related to managing Azure Data Factory (ADF) pipelines. I am documenting below the PowerShell commands I frequently use to achieve the results I expect. Retrieve pipelines from a given factory
|
1 |
$pipelines = Get-AzureRmDataFactoryPipeline -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup |
If you have multiple factories, you can append the results of another call in the same result variable
|
1 |
$pipelines += Get-AzureRmDataFactoryPipeline -DataFactoryName MyOtherFactory -ResourceGroupName MyOtherResourceGroup |
Similarly, there are commands to retrieve datasets and linked services
|
1 2 |
$datasets = Get-AzureRMDataFactoryDataSet -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup $linkedServices = Get-AzureRmDataFactoryLinkedService -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup |
The hierarchy of ADF pipelines goes as follow (there may be variations to this pattern that I am not aware): Factory → Pipelines → Activities → DataSets → LinkedServices Looping on the hierarchy:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
$pipelines | ForEach-Object { $pipelineName = $_.PipelineName $activities = $_.Properties.Activities $activities | ForEach-Object { $activityName = $_.Name $inputs = $_.Inputs $outputs = $_.Outputs $inputs | ForEach-Object { # and so on } } } |
Much of the information retrieved by these commands are in complex types. Most of them contains a Type property […]