Technology – Software Architect

Working with Azure Data Factory Pipelines

Posted on 2017-12-152017-12-15

I am often tasked with automating tasks related to managing Azure Data Factory (ADF) pipelines. I am documenting below the PowerShell commands I frequently use to achieve the results I expect. Retrieve pipelines from a given factory

$pipelines = Get-AzureRmDataFactoryPipeline -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup

1	$pipelines = Get-AzureRmDataFactoryPipeline -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup

If you have multiple factories, you can append the results of another call in the same result variable

$pipelines += Get-AzureRmDataFactoryPipeline -DataFactoryName MyOtherFactory -ResourceGroupName MyOtherResourceGroup

1	$pipelines += Get-AzureRmDataFactoryPipeline -DataFactoryName MyOtherFactory -ResourceGroupName MyOtherResourceGroup

Similarly, there are commands to retrieve datasets and linked services

$datasets = Get-AzureRMDataFactoryDataSet -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup
$linkedServices = Get-AzureRmDataFactoryLinkedService -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup

1 2	$datasets = Get-AzureRMDataFactoryDataSet -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup $linkedServices = Get-AzureRmDataFactoryLinkedService -DataFactoryName MyFactory -ResourceGroupName MyResourceGroup

The hierarchy of ADF pipelines goes as follow (there may be variations to this pattern that I am not aware): Factory → Pipelines → Activities → DataSets → LinkedServices Looping on the hierarchy:

$pipelines | ForEach-Object {
    $pipelineName = $_.PipelineName
    $activities = $_.Properties.Activities
    $activities | ForEach-Object { 
        $activityName = $_.Name
        $inputs = $_.Inputs
        $outputs = $_.Outputs

        $inputs | ForEach-Object {
            # and so on
        }
    }
}

$pipelines | ForEach-Object {

$pipelineName = $_.PipelineName

$activities = $_.Properties.Activities

$activities | ForEach-Object {

$activityName = $_.Name

$inputs = $_.Inputs

$outputs = $_.Outputs

$inputs | ForEach-Object {

# and so on

}

Much of the information retrieved by these commands are in complex types. Most of them contains a Type property […]

Technology

Service Health Check

Posted on 2017-04-012017-04-01

I spent some time back in Feb 2017 to implement a service health check for our WCF services. The idea was to attach a new endpoint for every real service endpoint, where the new endpoint would give a standard health check information. So, if you knew the service endpoint, the health check endpoint would be easily discovered from that. Microsoft has just released the Azure Monitor that provides monitoring telemetry for all cloud resources. Another tool for the toolbox! https://azure.microsoft.com/en-us/blog/announcing-the-general-availability-of-azure-monitor/

Technology

Service Fabric Introduction

Posted on 2017-03-20

Service Fabric is a distributed system platform that abstracts out much of the complexity of packaging, deploying and managing scale and reliable applications. There are plenty of cool online material that provide introduction on Service Bus, including videos. I will not repeat them myself, but instead provide links and a brief description about them. https://docs.microsoft.com/en-us/azure/service-fabric/ Landing page for SF documentation https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-overview 7-min long introduction with links to a couple of videos https://azure.microsoft.com/en-us/resources/videos/index/?services=service-fabric Lots of videos about SF https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-create-your-first-application-in-visual-studio Hello World!?! No, finally somebody created a simple example that is not called Hello World.

Technology

Hashing Source Code Files

Posted on 2017-03-18

You can match source files and binaries by hashing the source code: “might be useful for tracking, processing and controlling source code files that have a strong linkage to the specific executable files, resulting in a higher level of end-user confidence in the executable files.” https://msdn.microsoft.com/en-us/magazine/mt795185

Technology

Temporal Tables – Soft Updates

Posted on 2017-03-182017-03-18

Just read an interesting article about soft updates. It seems that SQL Server 2016 has a new feature to enable temporal tables: SYSTEM_VERSIONING. For those who don’t know, temporal databases maintain multiple copies of each piece of data instead of only storing the data that’s true at the current time. Why do I want this: this approach maintain the entire history of the data, including updates and delete information. Data mining anyone? Auditing anyone? How have we done this in the past? I have seen DBAs creating tools to maintain the database schema. Those tools generate SQL scripts that are executed in SQL Server to create and to update the tables. In addition, the tools would create an aggregate table (appended with _History […]

Technology

General Mobile Development Reading List

Posted on 2017-03-17

http://www.discoversdk.com/blog/ionic-2-vs-reactnative-vs-nativescript A comparison among Ionic 2, NativeScript and ReactScript from Feb 2017. Pros and cons of each tool is available on the page. https://www.codementor.io/fmcorz/react-native-vs-ionic-du1087rsw This is a slightly older article (Aug 2016) that compares React Native and Ionic, focusing on development process, testing, community, etc. The author doesn’t choose the better tool, but instead indicates his favorite: React Native. https://medium.com/react-id/ionic-framework-hybrid-app-vs-react-native-4facdd93f690#.zek9fvi1g This article also compares Ionic and Reatic Native, but focus more on app performance and size. The author indicates React Native as a better choice.

Technology

Async/Await best practices

Posted on 2017-03-17

Here are some best practices on async/await programing model based on reading from: Async/Await – Best Practices in Asynchronous Programming TPL and async/await Best Practices for the Busy Developer Use ‘async Task’ or ‘async Task<T>’ instead of ‘async void’. Why? Primarily because of error-handling semantics. When using ‘async void’, any exception thrown will be raised directly on the SynchronizationContext that was active when the async void method started. When using ‘async Task’, exception is captured and plated on the Task object. The exception is asynchronous event handlers, which must regturn void (ex: ICommand.Execute implementations)

async Task MyMethodAsync()
{
  // Do asynchronous work.
  await Task.Delay(1000);
}

async Task MyMethodAsync()

{

// Do asynchronous work.

await Task.Delay(1000);

}

Async code works best if it goes all the way from bottom up (or top down). That means that you shouldn’t mix synchronous and asynchronous […]

Technology

WPF UI Responsiveness

Posted on 2017-03-172017-03-17

All window objects are associated with the dispatcher where it has been created. Golden rules: Any UI element must be touched from the same UI thread that created it Don’t block the UI thread How to execute slow work then? In summary, perform slow work asynchronously update the UI using the Dispatcher.BeginInvoke Example:

ThreadStart ts = delegate
{
   DoSlowWork();
   Dispatcher.BeginInvoke(DispatcherPriority.Normal, (EventHandler) delegate
   {
      DoUIUpdate();
   }, null, null);

};
ts.BeginInvoke(delegate(IAsyncResult iar) {ts.EndInvoke(iar);}, null );

ThreadStart ts = delegate

{

DoSlowWork();

Dispatcher.BeginInvoke(DispatcherPriority.Normal, (EventHandler) delegate

{

DoUIUpdate();

}, null, null);

};

ts.BeginInvoke(delegate(IAsyncResult iar) {ts.EndInvoke(iar);}, null );

DispatcherPriority list …. Windows forms way:

SynchronizationContext syncContext = SynchronizationContext.Current;
 
ThreadStart ts = delegate
{
   DoSlowWork();
   syncContext.Post(delegate
   {
       DoUIUpdate();
   }, null);
};
ts.BeginInvoke(delegate(IAsyncResult iar) {ts.EndInvoke(iar);}, null );

SynchronizationContext syncContext = SynchronizationContext.Current;

ThreadStart ts = delegate

{

DoSlowWork();

syncContext.Post(delegate

{

DoUIUpdate();

}, null);

};

ts.BeginInvoke(delegate(IAsyncResult iar) {ts.EndInvoke(iar);}, null );

Technology

Deploying app.config transformation through NuGet packages

Posted on 2015-12-302015-12-30

There is a not-very-well-documented, and in my opinion limited, library from Microsoft called XML Document Transformation (XDT) that can be used to transform XML files by using XML attributes and a transformation file. The syntax reference information can be found here. The library itself is also distributed via NuGet. Many online articles have discussed the *.config transformation, including Custom web.config transforms and merge, Using web.config transforms with app.config files and XDT (web.config) Transforms in non-web projects. Additionally, deploying XDT in NuGet packages have also been discussed in a few articles, such as How to use XDT in NuGet – Examples and Facts, and Configuration File and Source Code Transformation. The limitations of the XDT are observed by the number of out-of-the-box verbs available to perform the transformation: Insert, […]

Technology

Setting the Thread.CurrentPrincipal inside an async method

Posted on 2015-12-202015-12-30

Earlier this week I spent more time than I wanted for trying to figure out why I was losing the Thread.CurrentPrincipal value after having set it inside an async method. My history goes a lot like the one described here. But it took me sometime to find that website on Google, which was retrieving many results for web applications but very few for desktop. I also found similar problem described by this post. In summary, I was working on authentication for a desktop application, using Open ID Connect and Identity Server. My calls to the user store are all based on WCF services, and I wanted to have the authentication done asynchronously to prevent my UI from locking up. According to […]

Share:

Share:

Share:

Share:

Share:

Share:

Share:

Share:

Share:

Share: