How to deploy Azure Logic Apps using Azure DevOps

If you’re looking answer for the following:

  • How to use Azure DevOps to deploy Azure Logic Apps?
  • How to do Continuous deployment of Azure Logic Apps using Azure DevOps?
  • How to develop Azure Logic Apps using Visual Studio and deploy it using AzureDevops?

You’re in the right place.

In this demonstration, I’ll be using the following tools / services

  • Visual Studio 2017 with Logic Apps Tools 
  • AzureDevOps (as of writing, they’re changing the product every 3 weeks)

Walkthough

  • Create Azure Logic Apps using Visual Studio 2017
    • Create multiple parameters per environment
  • Azure DevOps setup
    • Setup of AzureDevOps Service Connections
    • Setup of AzureDevOps Pipelines
    • Setup of AzureDevOps Release

Create Azure Logic Apps using Visual Studio 2017

  1. Open Visual Studio and connect to your AzureDevOps Instance
  2.  Once connected, Click File -> New Project -> Cloud -> Azure Resource Group

LogicAppProject

3. Select Logic app template and click next.

LogicAppTemplate

4. Open the logic app by Right Clicking the LogicApp.json -> Select Open With Logic App Designer.

LogicAppOpenDesigner

5. It will prompt the Logic App Properties, from here you’ll have to select the Subscription and resource group.

LogicAppSubscription

6. Develop the Logic App. You can copy from the list of template already available.

7. Create a copy of parameters file. In this example, I’ll create 3 different parameters files (dev, test and production). In the parameters file, i’ll use different logic app name per environment. The param file will be used later on in the AzureDevOps Release.

LogicAppsParams

8. Checkin the source code to Azure DevOps and later on we will link the Azure Pipeline to the source control.

Once this is done, we’re ready to configure and deploy using Azure DevOps.

Azure DevOps Setup

  1. Setup the Service Connection, this is needed when we configure the Release.
  2. Go to AzureDevOps project -> Click the Configure -> Pipelines -> Service Connections -> New Service Connection. Select Add an Azure Resource Manager service connectionAzureDevOpsServiceConnection
  3. Add a connection name. For the first example you can name it Azure Development. Create 2 more connections for Test and Production. In all cases, make sure to select a different resrouce group.
  4. Setup the Azure DevOps Pipelines, nothing special just do a Build of solution and Publish of artifact.AzurePipelineSetup
  5. Setup the Azure DevOps Release, overall it looks like this. and the configuration is repeating every stage (see #7). AzureReleaseView
  6. Development Stage Setup. You only need the Create Or Update Resource Group. Select the correct subscription (that we created in #1) and the template parameters (dev.parameters.json = Development)AzureReleaseSetup
  7. Clone the Development Stage to Test and Production.
  8. Only thing needs to be changed is the Subscription and the parameters.json file.

 

Testing

  1. Provided that you’ve setup the pipeline to do Continuous integration (Triggers-> Enable Continous integration). The moment you make changes to logic app and checkin the code. The Pipeline will be triggered.
  2. Once Pipeline succeeded, you can create a Release from it. This is how it looks like:AzureReleaseSuccesful

 

References:

http://www.integrationusergroup.com/continuous-integration-logic-apps-using-team-foundation-team-services/

 

 

 

 

 

 

 

 

 

Advertisements

BizTalk Map – Eliminate duplicate records using functoid

Recently there’s a requirement to eliminate the duplicate records in the output using BizTalk map.

Brilliant solution can be found here:

I’ve added step by step on how to do it.

Overall solution it looks like this:

Duplicate

You’ll need 2 script functiods and Equal logical functoid

  1. Top functoid will contain the declaration of variable with following code:

System.Collections.Specialized.StringCollection uniqueIds = new System.Collections.Specialized.StringCollection() ;

2.  Another functoid will take in the ID of source messsage wherein the uniqueness will be checked. In this case, CustomerId as input. The following code is as follows:

public bool IsUniqueId(string id)
{
if (uniqueIds.Contains(id))
return false;
uniqueIds.Add(id);
return true;
}

3.  Equal logical functoid will take the output of the second script functoid and output should be map to the target record

 

This is my input message:

<ns0:MasterData xmlns:ns0=”http://BizTalk_Server_Project1.Order”&gt;
<Customer>
<CustomerId>1</CustomerId>
<CustomerName>Customer 1</CustomerName>
</Customer>
<Customer>
<CustomerId>1</CustomerId>
<CustomerName>Deplicate Customer</CustomerName>
</Customer>
<Customer>
<CustomerId>2</CustomerId>
<CustomerName>Customer 2</CustomerName>
</Customer>
</ns0:MasterData>

And this is the output:

<?xml version=”1.0″?>

<ns0:MasterData xmlns:ns0=”http://BizTalk_Server_Project1.Order”&gt;

<Customer><CustomerId>1</CustomerId>

<CustomerName>Customer 1</CustomerName></Customer>

-<Customer>

<CustomerId>2</CustomerId>

<CustomerName>Customer 2</CustomerName>

</Customer>

</ns0:MasterData>

 

Viola!!, without using xslt you can achieve same results.

Azure API Management – How to set basic authentication in SendRequest

Scenario: I have both username and password stored in name value configuration of API management but I need to use it in SendRequest policy.

Source Code:

<set-variable name=“userName” value=“{{username}}” />

<set-variable name=“password” value=“{{password}}” />

<set-variable name=“basicAuthDetails” value=”@{

var username = context.Variables.GetValueOrDefault<string>(“userName”);

var password = context.Variables.GetValueOrDefault<string>(“password”);

return System.Convert.ToBase64String(System.Text.Encoding.GetEncoding(“ISO-8859-1”).GetBytes(username + “:” + password));

}/>

<set-variable name=“jsonPayload” value=”@{

JObject transBody = new JObject();

//Add all json Property

transBody.Add(“test”, JToken.FromObject(new[]

{

 “test data”

}));

}/>

<send-request mode=“new” response-variable-name=“var” ignore-error=“false”>

<set-url>@{

        var url = context.Api.ServiceUrl+ “/{someURL}”;

       return url;

}</set-url>

<set-method>POST</set-method>

<set-header name=“Authorization” exists-action=“override”>

<value>@(context.Variables.GetValueOrDefault<string>(“basicAuthDetails”))</value>

</set-header>

<set-header name=“Content-Type” exists-action=“override”>

<value>application/json</value>

</set-header>

<set-body template=“none”>@(context.Variables.GetValueOrDefault<string>(“jsonPayload”))</set-body>

</send-request>

 

That is how you can set basic authentication in SendRequest in Azure API Management.

Azure API Management Policy – Asynchronous API as Synchronous API

Below is the link to reference/examples of using Azure API management policy

There’s an example there on how to Mask Asynchronous calls as Synchronous API however for my requirement it’s not enough.

The target API behaves as follow:

  1. All operations are POST and asynchronous.
  2. To get the results of any operation a second API needs to be called and it’s expecting the transaction id.
  3. API expects a payload in command manner (args parameter) instead of proper JSON name/value pair.

Requirement: API should return the results in synchronous manner.

Solution:

API Callout Policy in API Management.

Steps: 

  1. Create a GET operation and via policy rewrite the operation to POST
  2. Create a JSON transformation logic in the inbound policy and execute the backend API.
  3. Create a JSON transformation logic in outbound policy and add a retry policy to execute the second API  that returns the result passing the transaction id from #2
  4. Return the results from second API.

Policy Code:

<policies>
<inbound>
<base />
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<!-- Save authorization data to variable, to be used in outbound status request -->
<set-variable name="authorization" value="@(context.Request.Headers.GetValueOrDefault("Authorization"))" />
<!-- 
Construct the payload
-->
<set-body>@{
var paramFromRequest = Uri.UnescapeDataString(context.Request.OriginalUrl.Query.GetValueOrDefault("paramFromRequest"));
JObject transBody = new JObject();
transBody.Add("source", 
     new JObject
     {
         {"someproperty", "fixvalue"},
         {"someproperty2", "fixvalue2"},
     });

//Add all json properties as arg
transBody.Add("args", JToken.FromObject(new[] 
{ 
      paramFromRequest
}));
return transBody.ToString();
}</set-body>
<trace source="debug">@(context.Request.Body.As<string>(true))</trace>
<!--Remove additional query string -->
<set-query-parameter name="paramFromRequest" exists-action="delete" />
<set-method>POST</set-method>
<rewrite-uri template="{{FIRSTBACKENDAPI_URL}}" />
</inbound>
<backend>
<base />
</backend>
<outbound>
<base />
<!-- Get the Key from response -->
<set-variable name="jsonPayload" value="@{ 
JObject inBody = context.Response.Body.As<JObject>();
var transactionId= inBody["transactionId"].ToString(); 
string jsonString = "{\"transactionId\":\"_transactionId\" }";
var json = jsonString.Replace("_transactionId", transactionId);
return json;
}" />
<retry condition="@( Convert.ToBoolean(context.Variables.GetValueOrDefault<JObject>("status")["{{PROPERTYNAME IN JSON OBJECT RESPONSE}}"]))" count="10" interval="1">
<!-- Call Status API to get the results-->
<send-request mode="new" response-variable-name="var" ignore-error="false">
<set-url>@{ 
var url = context.Api.ServiceUrl+ "{{SECONDBACKENDAPI_URL}}";
return url;
}</set-url>
<set-method>POST</set-method>
<set-header name="Authorization" exists-action="override">
<value>@(context.Variables.GetValueOrDefault<string>("authorization"))</value>
</set-header>
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-body template="none">@(context.Variables.GetValueOrDefault<string>("jsonPayload"))</set-body>
</send-request>
<set-variable name="results" value="@(((IResponse)context.Variables["var"]).Body.As<JObject>())" />
</retry>
<return-response>
<set-body template="none">@((context.Variables.GetValueOrDefault<JObject>("results")["{{PROPERTYNAME IN JSON OBJECT THAT CONTAINS THE RESULT}}"].ToString()))</set-body>
</return-response>
</outbound>
<on-error>
<base />
</on-error>
</policies>

 

Querying Azure SQL Database using Azure Functions 2.0 to return JSON data

The guide below shows how you can easily query Azure SQL Database using Azure Functions.

I have to admit, I have to do multiple google search and combine it for a working solution.

Challenges:

  1. How to get SQL connectionString from Azure Function settings. https://docs.microsoft.com/en-us/azure/azure-functions/functions-scenario-database-table-cleanup
  2. How to convert the sql results to JSON.  https://stackoverflow.com/questions/5083709/convert-from-sqldatareader-to-json

Steps:

    • 1. Create a new Function with HTTP trigger
    • 2. Add a new file called serialize.csx with following contents below. This will convert the SQL rows to a JSON like data.
using System.Text;
using System.Data;
using System.Linq;
using System.Configuration;
using System.Data.SqlClient;
using System.Collections.Generic;
using System.Collections;
public static IEnumerable<Dictionary<string, object>> Serialize(SqlDataReader reader)
{
var results = new List<Dictionary<string, object>>();
var cols = new List<string>();
for (var i = 0; i < reader.FieldCount; i++)
{
var colName = reader.GetName(i);
var camelCaseName = Char.ToLowerInvariant(colName[0]) + colName.Substring(1);
cols.Add(camelCaseName);
}

while (reader.Read())
results.Add(SerializeRow(cols, reader));

return results;
}
private static Dictionary<string, object> SerializeRow(IEnumerable<string> cols,
SqlDataReader reader) {
var result = new Dictionary<string, object>();
foreach (var col in cols)
result.Add(col, reader[col]);
return result;
}

 

3. In the run.csx, paste the following code. This will query the Azure SQL database and returns the data.

#r "Newtonsoft.Json"
#load "serialize.csx"

using System.Net;

using Microsoft.AspNetCore.Mvc;

using Microsoft.Extensions.Primitives;

using Newtonsoft.Json;

using System.Text;

using System.Data;

using System.Linq;

using System.Configuration;

using System.Data.SqlClient;

using System.Collections.Generic;

public static async Task<HttpResponseMessage> Run(HttpRequest req, ILogger log)

{

log.LogInformation("C# HTTP trigger function processed a request.");

string name = req.Query["name"];

string json =" ";

try

{

var str = Environment.GetEnvironmentVariable("<ConnectionStringName in appSettings>");




using(SqlConnection conn =new SqlConnection(str))

{

using(SqlCommand cmd =new SqlCommand())

{

SqlDataReader dataReader;

cmd.CommandText = "<SQL QUERY HERE>";

cmd.CommandType = CommandType.Text;

cmd.Connection = conn;

conn.Open();

dataReader = cmd.ExecuteReader();

var r = Serialize(dataReader);

json = JsonConvert.SerializeObject(r, Formatting.Indented);

}

}

}

catch(SqlException sqlex)

{

log.LogInformation(sqlex.Message);

log.LogInformation(sqlex.ToString());

returnnew HttpResponseMessage(HttpStatusCode.BadRequest)

{

Content = new StringContent(JsonConvert.SerializeObject($"The following SqlException happened: {sqlex.Message}"), Encoding.UTF8, "application/json")

};

}

catch(Exception ex)

{

log.LogInformation(ex.Message);

log.LogInformation(ex.ToString());

returnnew HttpResponseMessage(HttpStatusCode.BadRequest)

{

Content = new StringContent(JsonConvert.SerializeObject($"The following SqlException happened: {ex.Message}"), Encoding.UTF8, "application/json")

};

}

returnnew HttpResponseMessage(HttpStatusCode.OK)

{

Content = new StringContent(json, Encoding.UTF8, "application/json")

};

}
This shows how you can easily query the Azure SQL Database and return the data JSON in few minutes.
Perhaps next steps is to deploy this Azure Function into API management.

 

SAP – How to generate an XSD/IDoc Schema

Below are the steps on how to generate an XSD Schema / IDoc Schema from SAP.

1. In the SAP System in which the IDoc is defined, call transaction WE60 (IDoc Documentation).

2. Enter the name of the IDoc in the Basic type field (or the Enhancement field if your IDoc is an Enhancement of a standard SAP IDoc).

3. Choose menu Documentation, XML Schema.

4. Respond Yes, if prompted to ‘Generate Documentation for Unicode File’.

5. On the resulting screen containing the XSD of the IDoc, choose menu XML, Download.

SSIS 2012 to SSIS 2016 Migration – Arithmetic operation resulted in an overflow

We’ve recently upgraded our SSIS 2012 to SSIS 2016, during the package upgrade it goes without issue. In Visual Studio, there’s no problem running it, however when we use the SSIS runtime we’ve encountered the following error:

“Arithmetic operation resulted in an overflow.”. Possible failure reasons: Problems with the query, “ResultSet” property not set correctly, parameters not set correctly, or connection not established correctly.

The Execute method on the task returned error code 0x80131516 (Arithmetic operation resulted in an overflow.). The Execute method must succeed, and indicate the result using an “out” parameter.

To solve this issue,  I’ve recreated the SQL Task and viola no error!.