December 2017 – Software Architect

Working with Azure Data Factory Pipelines

Posted on 2017-12-152017-12-15

I am often tasked with automating tasks related to managing Azure Data Factory (ADF) pipelines. I am documenting below the PowerShell commands I frequently use to achieve the results I expect. Retrieve pipelines from a given factory

$pipelines = Get-AzureRmDataFactoryPipeline -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup

1	$pipelines = Get-AzureRmDataFactoryPipeline -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup

If you have multiple factories, you can append the results of another call in the same result variable

$pipelines += Get-AzureRmDataFactoryPipeline -DataFactoryName MyOtherFactory -ResourceGroupName MyOtherResourceGroup

1	$pipelines += Get-AzureRmDataFactoryPipeline -DataFactoryName MyOtherFactory -ResourceGroupName MyOtherResourceGroup

Similarly, there are commands to retrieve datasets and linked services

$datasets = Get-AzureRMDataFactoryDataSet -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup
$linkedServices = Get-AzureRmDataFactoryLinkedService -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup

1 2	$datasets = Get-AzureRMDataFactoryDataSet -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup $linkedServices = Get-AzureRmDataFactoryLinkedService -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup

The hierarchy of ADF pipelines goes as follow (there may be variations to this pattern that I am not aware): Factory → Pipelines → Activities → DataSets → LinkedServices Looping on the hierarchy:

$pipelines | ForEach-Object {
    $pipelineName = $_.PipelineName
    $activities = $_.Properties.Activities
    $activities | ForEach-Object { 
        $activityName = $_.Name
        $inputs = $_.Inputs
        $outputs = $_.Outputs

        $inputs | ForEach-Object {
            # and so on
        }
    }
}

$pipelines | ForEach-Object {

$pipelineName = $_.PipelineName

$activities = $_.Properties.Activities

$activities | ForEach-Object {

$activityName = $_.Name

$inputs = $_.Inputs

$outputs = $_.Outputs

$inputs | ForEach-Object {

# and so on

}

Much of the information retrieved by these commands are in complex types. Most of them contains a Type property […]

Share: