Technology

Working with Azure Data Factory Pipelines

I am often tasked with automating tasks related to managing Azure Data Factory (ADF) pipelines. I am documenting below the PowerShell commands I frequently use to achieve the results I expect.

  1. Retrieve pipelines from a given factory

    If you have multiple factories, you can append the results of another call in the same result variable
  2. Similarly, there are commands to retrieve datasets and linked services

The hierarchy of ADF pipelines goes as follow (there may be variations to this pattern that I am not aware):

Factory → Pipelines → Activities → DataSets → LinkedServices

Looping on the hierarchy:

Much of the information retrieved by these commands are in complex types. Most of them contains a Type property and a Properties property. The information from complex types can be flattened out using the following:

List unique type values:

Group type values:

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.