Querying Azure SQL Database using Azure Functions 2.0 to return JSON data

The guide below shows how you can easily query Azure SQL Database using Azure Functions.

I have to admit, I have to do multiple google search and combine it for a working solution.

Challenges:

  1. How to get SQL connectionString from Azure Function settings. https://docs.microsoft.com/en-us/azure/azure-functions/functions-scenario-database-table-cleanup
  2. How to convert the sql results to JSON.  https://stackoverflow.com/questions/5083709/convert-from-sqldatareader-to-json

Steps:

    • 1. Create a new Function with HTTP trigger
    • 2. Add a new file called serialize.csx with following contents below. This will convert the SQL rows to a JSON like data.
using System.Text;
using System.Data;
using System.Linq;
using System.Configuration;
using System.Data.SqlClient;
using System.Collections.Generic;
using System.Collections;
public static IEnumerable<Dictionary<string, object>> Serialize(SqlDataReader reader)
{
var results = new List<Dictionary<string, object>>();
var cols = new List<string>();
for (var i = 0; i < reader.FieldCount; i++)
{
var colName = reader.GetName(i);
var camelCaseName = Char.ToLowerInvariant(colName[0]) + colName.Substring(1);
cols.Add(camelCaseName);
}

while (reader.Read())
results.Add(SerializeRow(cols, reader));

return results;
}
private static Dictionary<string, object> SerializeRow(IEnumerable<string> cols,
SqlDataReader reader) {
var result = new Dictionary<string, object>();
foreach (var col in cols)
result.Add(col, reader[col]);
return result;
}

 

3. In the run.csx, paste the following code. This will query the Azure SQL database and returns the data.

#r "Newtonsoft.Json"
#load "serialize.csx"

using System.Net;

using Microsoft.AspNetCore.Mvc;

using Microsoft.Extensions.Primitives;

using Newtonsoft.Json;

using System.Text;

using System.Data;

using System.Linq;

using System.Configuration;

using System.Data.SqlClient;

using System.Collections.Generic;

public static async Task<HttpResponseMessage> Run(HttpRequest req, ILogger log)

{

log.LogInformation("C# HTTP trigger function processed a request.");

string name = req.Query["name"];

string json =" ";

try

{

var str = Environment.GetEnvironmentVariable("<ConnectionStringName in appSettings>");




using(SqlConnection conn =new SqlConnection(str))

{

using(SqlCommand cmd =new SqlCommand())

{

SqlDataReader dataReader;

cmd.CommandText = "<SQL QUERY HERE>";

cmd.CommandType = CommandType.Text;

cmd.Connection = conn;

conn.Open();

dataReader = cmd.ExecuteReader();

var r = Serialize(dataReader);

json = JsonConvert.SerializeObject(r, Formatting.Indented);

}

}

}

catch(SqlException sqlex)

{

log.LogInformation(sqlex.Message);

log.LogInformation(sqlex.ToString());

returnnew HttpResponseMessage(HttpStatusCode.BadRequest)

{

Content = new StringContent(JsonConvert.SerializeObject($"The following SqlException happened: {sqlex.Message}"), Encoding.UTF8, "application/json")

};

}

catch(Exception ex)

{

log.LogInformation(ex.Message);

log.LogInformation(ex.ToString());

returnnew HttpResponseMessage(HttpStatusCode.BadRequest)

{

Content = new StringContent(JsonConvert.SerializeObject($"The following SqlException happened: {ex.Message}"), Encoding.UTF8, "application/json")

};

}

returnnew HttpResponseMessage(HttpStatusCode.OK)

{

Content = new StringContent(json, Encoding.UTF8, "application/json")

};

}
This shows how you can easily query the Azure SQL Database and return the data JSON in few minutes.
Perhaps next steps is to deploy this Azure Function into API management.

 

Advertisements

SSIS 2012 to SSIS 2016 Migration – Arithmetic operation resulted in an overflow

We’ve recently upgraded our SSIS 2012 to SSIS 2016, during the package upgrade it goes without issue. In Visual Studio, there’s no problem running it, however when we use the SSIS runtime we’ve encountered the following error:

“Arithmetic operation resulted in an overflow.”. Possible failure reasons: Problems with the query, “ResultSet” property not set correctly, parameters not set correctly, or connection not established correctly.

The Execute method on the task returned error code 0x80131516 (Arithmetic operation resulted in an overflow.). The Execute method must succeed, and indicate the result using an “out” parameter.

To solve this issue,  I’ve recreated the SQL Task and viola no error!.

 

 

SSIS – Use timestamp to detect changes

Last week, a solution was required to build an SSIS solution that can pick up all changes from a database using the versionnumber (timestamp) and sync the changes to an external database using the timestamp of the database. This is quite easy to achieve by:

  1. Step 1, retrieve the previous timestamp. Storing the previous timestamp can be easily achieved by create a file using File.WriteAllText method.

 Dim runLastFile = System.IO.Path.Combine(currentPath, “LastRun.txt”)     System.IO.File.WriteAllText(runLastFile, lastRun)

I know it’s in VB but you can easily convert it to C#, since it’s an old solution i didn’t bother to convert it. If the file is not yet existing, you need to create a logic to get the lowest timestamp. Like this:

declare
@minVersion binary(10),
@minVersion_str nvarchar(42)

SET @minVersion = ( SELECT MIN(timestamp fieldName) FROM {sourceTable} )
SET @minVersion_str = upper(sys.fn_varbintohexstr(@minVersion))
SELECT @minVersion_str as LowestVersion

2. Step 2, get the current timestamp from the source database by using MIN_ACTIVE_ROWVERSION, the challenge however is to store this as string/text so the SQL query can be constructed later on in the process. After 15 minutes of googling I was able to find it somewhere (credit to the forum, lost the link).

declare
@currVersion binary(10),
@currVersion_str nvarchar(42)

SET @currVersion = MIN_ACTIVE_ROWVERSION()
SET @currVersion_str = upper(sys.fn_varbintohexstr(@currVersion))
SELECT @currVersion_str as CurrentVersion

3. Step 3, construct the SQL query using the previous timestamp. Something like this:

                         SELECT {sourceColumn}
FROM {sourceTable}
WHERE {timestamp fieldName}  >= @previousTimeStamp

4. Step 4, if the process is successful don’t forget to store the current timestamp.

5. Step 5, start from step 1.

[Powershell] Executing an sql query and returning a disconnected data using Powershell

Below is a script that will execute an sql query and returns a disconnected data (DataTable) using Powershell.

<#
.SYNOPSIS
	Queries a data source and returns a disconnected data (DataTable)

.DESCRIPTION
	Queries a data source and returns a disconnected data (DataTable)

.NOTES
	File Name: Get-DbDataTable.ps1
	Author: Randy Aldrich Paulo

.EXAMPLE
	Get-DbDataTable "Data Source=.;Integrated Security=SSPI;Initial Catalog=DatabaseName" "SELECT TOP 10 * From Table1"
#>

function Get-DbDataTable
(
	[string]$connectionString,
	[string]$query,
	[System.Data.CommandType]$commandType=[System.Data.CommandType]::Text,
	[string]$providerName = "System.Data.SqlClient"	
)
{
	#Initialize
	$databaseFactory = [System.Data.Common.DbProviderFactories]::GetFactory($providerName)
	$dbConnection = $databaseFactory.CreateConnection()
	$dbConnection.ConnectionString = $connectionString

	#Create Command
	$dbCommand = $databaseFactory.CreateCommand()
	$dbCommand.Connection = $dbConnection
	$dbCommand.CommandText =$query
	$dbCommand.CommandType = $commandType

	#Create Data Adapter
	$dbAdapter = $databaseFactory.CreateDataAdapter();
	$dbAdapter.SelectCommand = $dbCommand

	$dataTable = New-Object System.Data.DataTable

	try
	{
		#Fill with Results
		$dbAdapter.Fill($dataTable)
	}
	catch { 
		throw 
		}
	finally { 
			if($dbConnection.ConnectionState -ne [System.Data.ConnectionState]::Closed) 
			{ 
				$dbConnection.Close() 
			}
		}
	return $dataTable
}

[Powershell] Set user permission in SSRS Item (SQL Reporting Server) using powershell

Combining both (article1 and article2), i was able to come up with a powershell script to set user permission in SQL Reporting Services (SSRS)

<#
.SYNOPSIS
	Set user permissions in SQL Reporting Services using Web Service

.DESCRIPTION
	Set user permissions in SQL Reporting Services using Web Service

.EXAMPLE
	Add-SSRSItemSecurity -webServiceUrl "http://[ServerName]/ReportServer/ReportService2005.asmx?WSDL" -itemPath "MyReportFolder" -groupUserName RPAULO\User1 -role Browser

.EXAMPLE
	Add-SSRSItemSecurity -url "http://[ServerName]/ReportServer/ReportService2005.asmx?WSDL" -itemPath "MyReportFolder" -u RPAULO\User1 -r "Content Manager"

#>
function Add-SSRSItemSecurity
(
	[Parameter(Position=0,Mandatory=$true)]
	[Alias("url")]
	[string]$webServiceUrl,

	[Parameter(Position=1,Mandatory=$true)]
	[Alias("path")]
	[string]$itemPath,
	
	[Parameter(Position=2,Mandatory=$true)]
	[Alias("u")]
	[string]$groupUserName,
	
	[Parameter(Position=3,Mandatory=$true)]
	[Alias("r")]
	[string]$role,
	
	[Parameter(Position=2)]
	[bool]$inherit=$true
)

{
	
	#Fix item path if not starting with /
	if(!$itemPath.StartsWith("/")) { $itemPath = "/" + $itemPath}
	
	#Create Proxy
	Write-Host "[Add-SSRSItemSecurity()] Creating Proxy, connecting to : $webServiceUrl"
	$ssrsProxy = New-WebServiceProxy -Uri $webServiceUrl -UseDefaultCredential
	
	$type = $ssrsProxy.GetType().Namespace;
	$policyType = "{0}.Policy" -f $type;
	$roleType = "{0}.Role" -f $type;
	
	Write-Host "[Add-SSRSItemSecurity()] Retrieving all existing policies."
	$policies = $ssrsProxy.GetPolicies($itemPath, [ref]$inherit);
	
	$a = 1;
	foreach($policy in $policies)
	{

		foreach($r in $policy.Roles)
		{
			$msg = "[Add-SSRSItemSecurity()]  Existing Policy # {0} Group Name: {1}, Role: {2}" -f $a, $policy.GroupUserName, $r.Name
			Write-Host $msg
		}
		$a+=1;
	}

	$msg = "[Add-SSRSItemSecurity()] Total Existing Policies: " + $policies.Length;
	Write-Host $msg
	
	$Policy = $policies | 
    Where-Object { $_.GroupUserName -eq $groupUserName } | 
    Select-Object -First 1
	
	if (-not $Policy) {
	    $Policy = New-Object ($policyType)
	    $Policy.GroupUserName = $GroupUserName
	    $Policy.Roles = @()
	    $Policies += $Policy
		$msg = "[Add-SSRSItemSecurity()] Adding new policy: '{0}'" -f $GroupUserName
		Write-Host $msg
	}

	$r = $Policy.Roles |
	    Where-Object { $_.Name -eq $role } |
	    Select-Object -First 1
	if (-not $r) {
	    $r = New-Object ($roleType)
	    $r.Name = $role
	    $Policy.Roles += $r
		$msg = "[Add-SSRSItemSecurity()] Adding new role: '{0}'" -f $role
		Write-Host $msg
	}
	
	#Set the policies
	$ssrsProxy.SetPolicies($itemPath,$policies);

}


[Powershell] How to Install/deploy SSRS (rdl files) using Powershell

Since Powershell v2.0 introduce the new cmdlet New-WebServiceProxy, it’s no longer needed to generate a class library to deploy/undeploy SSRS file (RDL), below are two powershell scripts Install-SSRSRDL and Uninstall-SSRSRDL that can be useful when doing an automated deployment or silent installation of rdl files to Reporting Services.

Install SSRS RDL Script:


<#
.SYNOPSIS
	Installs an RDL file to SQL Reporting Server using Web Service

.DESCRIPTION
	Installs an RDL file to SQL Reporting Server using Web Service

.NOTES
	File Name: Install-SSRSRDL.ps1
	Author: Randy Aldrich Paulo
	Prerequisite: SSRS 2008, Powershell 2.0

.PARAMETER reportName
	Name of report wherein the rdl file will be save as in Report Server.
	If this is not specified it will get the name from the file (rdl) exluding the file extension.

.PARAMETER force
	If force is specified it will create the report folder if not existing
	and overwrites the report if existing.

.EXAMPLE
	Install-SSRSRDL -webServiceUrl "http://[ServerName]/ReportServer/ReportService2005.asmx?WSDL" -rdlFile "C:\Report.rdl" -force

.EXAMPLE
	Install-SSRSRDL "http://[ServerName]/ReportServer/ReportService2005.asmx?WSDL" "C:\Report.rdl" -force

.EXAMPLE
	Install-SSRSRDL "http://[ServerName]/ReportServer/ReportService2005.asmx?WSDL" "C:\Report.rdl" -force -reportName "MyReport"

.EXAMPLE
	Install-SSRSRDL "http://[ServerName]/ReportServer/ReportService2005.asmx?WSDL" "C:\Report.rdl" -force -reportFolder "Reports" -reportName "MyReport"

#>
function Install-SSRSRDL
(
	[Parameter(Position=0,Mandatory=$true)]
	[Alias("url")]
	[string]$webServiceUrl,

	[ValidateScript({Test-Path $_})]
	[Parameter(Position=1,Mandatory=$true)]
	[Alias("rdl")]
	[string]$rdlFile,

	[Parameter(Position=2)]
	[Alias("folder")]
	[string]$reportFolder="",

	[Parameter(Position=3)]
	[Alias("name")]
	[string]$reportName="",

	[switch]$force
)
{
	$ErrorActionPreference="Stop"

	#Create Proxy
	Write-Host "[Install-SSRSRDL()] Creating Proxy, connecting to : $webServiceUrl"
	$ssrsProxy = New-WebServiceProxy -Uri $webServiceUrl -UseDefaultCredential
	$reportPath = "/"

	if($force)
	{
		#Check if folder is existing, create if not found
		try
		{
			$ssrsProxy.CreateFolder($reportFolder, $reportPath, $null)
			Write-Host "[Install-SSRSRDL()] Created new folder: $reportFolder"
		}
		catch [System.Web.Services.Protocols.SoapException]
		{
			if ($_.Exception.Detail.InnerText -match "[^rsItemAlreadyExists400]")
			{
				Write-Host "[Install-SSRSRDL()] Folder: $reportFolder already exists."
			}
			else
			{
				$msg = "[Install-SSRSRDL()] Error creating folder: $reportFolder. Msg: '{0}'" -f $_.Exception.Detail.InnerText
				Write-Error $msg
			}
		}

	}

	#Set reportname if blank, default will be the filename without extension
	if($reportName -eq "") { $reportName = [System.IO.Path]::GetFileNameWithoutExtension($rdlFile);}
	Write-Host "[Install-SSRSRDL()] Report name set to: $reportName"

	try
	{
		#Get Report content in bytes
		Write-Host "[Install-SSRSRDL()] Getting file content (byte) of : $rdlFile"
		$byteArray = gc $rdlFile -encoding byte
		$msg = "[Install-SSRSRDL()] Total length: {0}" -f $byteArray.Length
		Write-Host $msg

		$reportFolder = $reportPath + $reportFolder
		Write-Host "[Install-SSRSRDL()] Uploading to: $reportFolder"

		#Call Proxy to upload report
		$warnings = $ssrsProxy.CreateReport($reportName,$reportFolder,$force,$byteArray,$null)
		if($warnings.Length -eq $null) { Write-Host "[Install-SSRSRDL()] Upload Success." }
		else { $warnings | % { Write-Warning "[Install-SSRSRDL()] Warning: $_" }}
	}
	catch [System.IO.IOException]
	{
		$msg = "[Install-SSRSRDL()] Error while reading rdl file : '{0}', Message: '{1}'" -f $rdlFile, $_.Exception.Message
		Write-Error msg
	}
	catch [System.Web.Services.Protocols.SoapException]
	{
		$msg = "[Install-SSRSRDL()] Error while uploading rdl file : '{0}', Message: '{1}'" -f $rdlFile, $_.Exception.Detail.InnerText
		Write-Error $msg
	}

}

Uninstall SSRS RDL Script:


<#
.SYNOPSIS
	Uninstalls an RDL file from SQL Reporting Server using Web Service

.DESCRIPTION
	Uninstalls an RDL file from SQL Reporting Server using Web Service

.NOTES
	File Name: Uninstall-SSRSRDL.ps1
	Author: Randy Aldrich Paulo
	Prerequisite: SSRS 2008, Powershell 2.0

.EXAMPLE
	Uninstall-SSRSRDL -webServiceUrl "http://[ServerName]/ReportServer/ReportService2005.asmx?WSDL" -path "MyReport"

.EXAMPLE
	Uninstall-SSRSRDL -webServiceUrl "http://[ServerName]/ReportServer/ReportService2005.asmx?WSDL" -path "Reports/Report1"

#>
function Uninstall-SSRSRDL
(
	[Parameter(Position=0,Mandatory=$true)]
	[Alias("url")]
	[string]$webServiceUrl,

	[Parameter(Position=1,Mandatory=$true)]
	[Alias("path")]
	[string]$reportPath
)

{
	#Create Proxy
	Write-Host "[Uninstall-SSRSRDL()] Creating Proxy, connecting to : $webServiceUrl"
	$ssrsProxy = New-WebServiceProxy -Uri $webServiceUrl -UseDefaultCredential

	#Set Report Folder
	if(!$reportPath.StartsWith("/")) { $reportPath = "/" + $reportPath }

	try
	{

		Write-Host "[Uninstall-SSRSRDL()] Deleting: $reportPath"
		#Call Proxy to upload report
		$ssrsProxy.DeleteItem($reportPath)
		Write-Host "[Uninstall-SSRSRDL()] Delete Success."
	}
	catch [System.Web.Services.Protocols.SoapException]
	{
		$msg = "[Uninstall-SSRSRDL()] Error while deleting report : '{0}', Message: '{1}'" -f $reportPath, $_.Exception.Detail.InnerText
		Write-Error $msg
	}

}

Install-SSRSRDL Powershell Script
Uninstall-SSRSRDL Powershell Script

For permission check this blog: https://randypaulo.wordpress.com/2012/02/22/powershell-set-user-permission-in-ssrs-item-sql-reporting-server-using-powershell/

SSRS Chart Label Missing (X/Y Axis missing)

When designing an SSRS Report using Chart you may notice that some of the labels whether in X or Y axis are not being displayed, this is because it’s so smart that it detects what you actually want (not!). 

To display all labels follow the steps below:

Right click either the X or Y axis, Click Properties:

Set the Interval to 1:

Viola!