Key to successful BizTalk Deployment

It’s been a while since my last blog, but IMHO this is one of important blog that I’ve ever made that can be useful pointer for every aspiring developer / companies that will/already implement application integration. Below contains a list / summary of best practices I’ve acquired in my more or less 10 years working with BizTalk, SSIS, WCF and .NET integration (primarily Microsoft Technology) but this I think is also applicable to other technology.

Design / Development:

#1 K.I.S.S (Keep it simple, stupid). Don’t get me wrong, I like complex integrations, from  multiple message correlations, requirement to aggregate certain line items with certain condition in the message to having a direct binding with delivery notification set to transmitted (multiple send port / send port groups, this will result in some zombie messages). Anyone from novice to expert BizTalk Developer can build a solution, it can be done & solve  in different ways but the key here is providing a simple and effective solution for a complex integration.

#2 Isolation with maximum reusability. Isolation for me means that the application can be deployed without having too much dependency to other BizTalk artifacts that are shared. During the design it’s always a struggle on how to seperate artifacts, on which solution / projects it should be placed to avoid having to uninstall all referencing application before you can refresh the dependency (GACing works in some situation but not for all, you’ll be surprise how BizTalk deal with it internally). Once identifying or isolating a certain integration flow whenever there’s a new requirement, we should always check whether it fits on already existing solution (extend) without having to build a new integration flow. in this case we will have maximum reuse of existing solution.

#3 Consistency. Every solution that will be build should look as if it’s build / coded by one person. Having a naming convention on BizTalk artifacts, namespace, solutions, project & folder structures should be define first. Based on experience providing a proper name is always a challenge since if you define it incorrectly, it will be a costly mistake later on.

#4 Nuget – all common code / components / libraries should be referenced from a local nuget repository. Having a local nuget repository not only simplifies referencing dependencies but also saves you once you start using TFS Build Server (autorestore feature). This means that shared assemblies is not required to be checked in together with the source codes.

#5 .NET over XLST in complex mapping  Whenever there’s a requirement for complex mapping like grouping, aggregation, unique numbering over a certain combinations or constructing a hashtable in the mapping, I always go for .NET component. Using .NET component it is easier to maintain and tested (unit testing + mockup) vs doing everything in XSLT (in which can be forgotten in time).

Build / Deployment:

#1 Release Pipeline – It’s important to define strategy on how you would build and be able to do ‘Build once’ and deploy to multiple environments. In our case, part of our strategy is to minimize branching and always build from MAIN. In TFS, we’ve implemented a Gated Checkin which means that every checkin will trigger a release pipeline. During the checkin, a build will be triggerred (MSBuild and BTDF build), it will run all unit test and copies the binaries to drop folder (only If all steps are successful). From this point the Ops can just enter a command (application name) and environment and then the automated deployment will take over..

#2 Configuration over Code for normal deployment most of time you would see that a solution contains a powershell script for deployment, this is not bad but i think this only works on sample codes or tutorials but a problem in Enterprise deployment. For this you need a good deployment framework (compilation of scripts) to do the deployment and during the deployment preparation just replace the values in the templates like servernames, shared folders and etc.  I’ve posted some blogs earlier how to do this:

Centralize Powershell Repository:

https://randypaulo.wordpress.com/2012/02/27/powershell-centralize-powershell-script-modules-repository/

BTDF: System.OutOfMemoryException while deploying using Powershell Remoting

Recently when I tried to deploy a BizTalk solution packaged with BTDF (BizTalk Deployment Framework) in combination with Powershell Remoting I encountered this error:

Error: Failed to add resource(s).

Change requests failed for some resources.

BizTalkAssemblyResourceManager failed to complete end type change request.

Exception of type ‘System.OutOfMemoryException’ was thrown.

 

I checked the memory on the remote server and even restarted it but it didn’t help, I still encounter this error.

As it turns out, as well as I’ve suspected it’s the memory of the powershell. To increase the size of the memory, run the PowerShell Command prompt on the target server and enter the ff:

Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 512

Or set it even to higher.

 

 

[Powershell] How to Install/deploy SSRS (rdl files) using Powershell

Since Powershell v2.0 introduce the new cmdlet New-WebServiceProxy, it’s no longer needed to generate a class library to deploy/undeploy SSRS file (RDL), below are two powershell scripts Install-SSRSRDL and Uninstall-SSRSRDL that can be useful when doing an automated deployment or silent installation of rdl files to Reporting Services.

Install SSRS RDL Script:


<#
.SYNOPSIS
        Installs an RDL file to SQL Reporting Server using Web Service

.DESCRIPTION
        Installs an RDL file to SQL Reporting Server using Web Service

.NOTES
        File Name: Install-SSRSRDL.ps1
        Author: Randy Aldrich Paulo
        Prerequisite: SSRS 2008, Powershell 2.0

.PARAMETER reportName
        Name of report wherein the rdl file will be save as in Report Server.
        If this is not specified it will get the name from the file (rdl) exluding the file extension.

.PARAMETER force
        If force is specified it will create the report folder if not existing
        and overwrites the report if existing.

.EXAMPLE
        Install-SSRSRDL -webServiceUrl "http://[ServerName]/ReportServer/ReportService2005.asmx" -rdlFile "C:\Report.rdl" -force

.EXAMPLE
        Install-SSRSRDL "http://[ServerName]/ReportServer/ReportService2005.asmx" "C:\Report.rdl" -force

.EXAMPLE
        Install-SSRSRDL "http://[ServerName]/ReportServer/ReportService2005.asmx" "C:\Report.rdl" -force -reportName "MyReport"

.EXAMPLE
        Install-SSRSRDL "http://[ServerName]/ReportServer/ReportService2005.asmx" "C:\Report.rdl" -force -reportFolder "Reports" -reportName "MyReport"

#>
function Install-SSRSRDL
(
        [Parameter(Position=0,Mandatory=$true)]
        [Alias("url")]
        [string]$webServiceUrl,

        [ValidateScript({Test-Path $_})]
        [Parameter(Position=1,Mandatory=$true)]
        [Alias("rdl")]
        [string]$rdlFile,

        [Parameter(Position=2)]
        [Alias("folder")]
        [string]$reportFolder="",

        [Parameter(Position=3)]
        [Alias("name")]
        [string]$reportName="",

        [switch]$force
)
{
        $ErrorActionPreference="Stop"

        #Create Proxy
        Write-Host "[Install-SSRSRDL()] Creating Proxy, connecting to : $webServiceUrl"
        $ssrsProxy = New-WebServiceProxy -Uri $webServiceUrl -UseDefaultCredential
        $reportPath = "/"

        if($force)
        {
                #Check if folder is existing, create if not found
                try
                {
                        $ssrsProxy.CreateFolder($reportFolder, $reportPath, $null)
                        Write-Host "[Install-SSRSRDL()] Created new folder: $reportFolder"
                }
                catch [System.Web.Services.Protocols.SoapException]
                {
                        if ($_.Exception.Detail.InnerText -match "[^rsItemAlreadyExists400]")
                        {
                                Write-Host "[Install-SSRSRDL()] Folder: $reportFolder already exists."
                        }
                        else
                        {
                                $msg = "[Install-SSRSRDL()] Error creating folder: $reportFolder. Msg: '{0}'" -f $_.Exception.Detail.InnerText
                                Write-Error $msg
                        }
                }

        }

        #Set reportname if blank, default will be the filename without extension
        if($reportName -eq "") { $reportName = [System.IO.Path]::GetFileNameWithoutExtension($rdlFile);}
        Write-Host "[Install-SSRSRDL()] Report name set to: $reportName"

        try
        {
                #Get Report content in bytes
                Write-Host "[Install-SSRSRDL()] Getting file content (byte) of : $rdlFile"
                $byteArray = gc $rdlFile -encoding byte
                $msg = "[Install-SSRSRDL()] Total length: {0}" -f $byteArray.Length
                Write-Host $msg

                $reportFolder = $reportPath + $reportFolder
                Write-Host "[Install-SSRSRDL()] Uploading to: $reportFolder"

                #Call Proxy to upload report
                $warnings = $ssrsProxy.CreateReport($reportName,$reportFolder,$force,$byteArray,$null)
                if($warnings.Length -eq $null) { Write-Host "[Install-SSRSRDL()] Upload Success." }
                else { $warnings | % { Write-Warning "[Install-SSRSRDL()] Warning: $_" }}
        }
        catch [System.IO.IOException]
        {
                $msg = "[Install-SSRSRDL()] Error while reading rdl file : '{0}', Message: '{1}'" -f $rdlFile, $_.Exception.Message
                Write-Error msg
        }
        catch [System.Web.Services.Protocols.SoapException]
        {
                $msg = "[Install-SSRSRDL()] Error while uploading rdl file : '{0}', Message: '{1}'" -f $rdlFile, $_.Exception.Detail.InnerText
                Write-Error $msg
        }

}

Uninstall SSRS RDL Script:


<#
.SYNOPSIS
        Uninstalls an RDL file from SQL Reporting Server using Web Service

.DESCRIPTION
        Uninstalls an RDL file from SQL Reporting Server using Web Service

.NOTES
        File Name: Uninstall-SSRSRDL.ps1
        Author: Randy Aldrich Paulo
        Prerequisite: SSRS 2008, Powershell 2.0

.EXAMPLE
        Uninstall-SSRSRDL -webServiceUrl "http://[ServerName]/ReportServer/ReportService2005.asmx" -path "MyReport"

.EXAMPLE
        Uninstall-SSRSRDL -webServiceUrl "http://[ServerName]/ReportServer/ReportService2005.asmx" -path "Reports/Report1"

#>
function Uninstall-SSRSRDL
(
        [Parameter(Position=0,Mandatory=$true)]
        [Alias("url")]
        [string]$webServiceUrl,

        [Parameter(Position=1,Mandatory=$true)]
        [Alias("path")]
        [string]$reportPath
)

{
        #Create Proxy
        Write-Host "[Uninstall-SSRSRDL()] Creating Proxy, connecting to : $webServiceUrl"
        $ssrsProxy = New-WebServiceProxy -Uri $webServiceUrl -UseDefaultCredential

        #Set Report Folder
        if(!$reportPath.StartsWith("/")) { $reportPath = "/" + $reportPath }

        try
        {

                Write-Host "[Uninstall-SSRSRDL()] Deleting: $reportPath"
                #Call Proxy to upload report
                $ssrsProxy.DeleteItem($reportPath)
                Write-Host "[Uninstall-SSRSRDL()] Delete Success."
        }
        catch [System.Web.Services.Protocols.SoapException]
        {
                $msg = "[Uninstall-SSRSRDL()] Error while deleting report : '{0}', Message: '{1}'" -f $reportPath, $_.Exception.Detail.InnerText
                Write-Error $msg
        }

}

Install-SSRSRDL Powershell Script
Uninstall-SSRSRDL Powershell Script

For permission check this blog: https://randypaulo.wordpress.com/2012/02/22/powershell-set-user-permission-in-ssrs-item-sql-reporting-server-using-powershell/

BizTalk Deployment Framework (BTDF) Tutorial – Basic Instruction – Walkthrough

BizTalk Deployment Framework for BizTalk 2010 Walkthrough

Since it’s really hard to find a simple instruction on how to use the BizTalk Deployment Framework (BTDF) i come up with compiled simplified instruction to get you started.

BTDF Files

When you add the BizTalk Deployment Framework in your project these are the important files that you need to take note of:

Deployment.btdfproj – this is the main file that contains the configuration on how your installer will behave.

SettingsFileGenerator.xml – this is an excel template that you can open using Excel in which you will map the settings (binding information) per environment. These files: Exported_DevSettings.xml, Exported_LocalSettings.xml, Exported_ProdSettings.xml, Exported_TestSettings.xml will be automatically generated by BTDF during installation.

Update Deployment.btdfproj

Double click Deployment.btdfproj to open.

The first part is pretty much self-explanatory:



Debug
x86
1.0
MyTest
1.0
True
 False
True
True
True
False
False
True
True
True


Take note of the the last 3 settings, I set SkipIISReset since I’m not deploying anything on IIS and UsingMasterBindings(set to false initially, will be set to true later on) since I want to use a single binding file for all environments. How is it possible? Basically I would first manually configure the bt application (receive location/send ports) then I will export the binding (MasterBinding) and I will add place holders inside it so that the XmlPreProcessor (Part of BTDF) will substitute during installation.  Pretty cool huh?

Lets go to the last part:



   


   
       ..\..\$(ProjectName)\Shared Assemblies
   
    
       ..\..\$(ProjectName)\Shared Assemblies
    
    
       ..\..\$(ProjectName)\Source\MyTest.Schemas\bin\$(Configuration)
    

 ..\..\$(ProjectName)\Source\MyTest.Transforms\bin\$(Configuration)
 
 
..\..\$(ProjectName)\Source\MyTest.Orchestrations\bin\$(Configuration)    


This means that I would like to

– Restart the BizTalk Host: SSO_Host and Sending_Host after the installation

– I have a common .NET component named CommonComponent.dll and common schema named  CommonSchema.dll both on Shared Assemblies folder.

– I have Schema, Orchestration and a Map. I changed the name because normally for big solution you don’t want to put all schemas in the schemas project and so on. Sometimes it’s better to split it functionally, this example configuration show you how  you can do that.

Next Steps

1. Deploy the solution manually or create a dummy PortBindings.xml (must be valid) to deploy using BTDF then click Tools -> Deployment Framework for BizTalk -> Deploy BizTalk Solution

2. Manually configure the receive ports, send ports, orchestration bindings.

3.  Export the bindings and named it as PortBindingsKnownGood.xml

4Run ElementTunnel.exe
/i:PortBindingsKnownGood.xml
/x:adapterXPaths.txt
/o:PortBindingsMaster.xml
/decode

Both (ElementTunnel and adapterXPaths.txt) can be found under \Deployment Framework for BizTalk\5.0\Framework\DeployTools.

The generated PortBindingsMaster.xml now has selected nested XML fragments decoded into plain, unencoded XML, controlled by the XPath statements in AdapterXPaths.txt. (see BTDF FAQ)

5. Add the the generated binding file: PortBindingsMaster.xml to the solution.

Configure PortBindingsMaster.xml

1. Double click PortBindingsMaster.xml to open.

2. Now go to the Address section of ReceivePort or SendPort and add a placeholder on the place you want to substitute during installation. Here I want the ${BackupOrderPath} substituted during installation.


${BackupOrderPath}
C:\Orders\Backup\%MessageID%.xml

 Configure SettingsFileGenerator.xml 

Right click SettingsFileGenerator.xml -> open with MS Excel.

This is how it should look like:

Deployment

1. Open Deployment.btdfproj and update both UsingMasterBindings and ApplyXmlEscape to True

2. Click Tools-> Deployment Framework for BizTalk -> Build Server Deploy MSI. This will generate an MSI file under the Project Folder\Deployment\bin folder.

3. Double click the MSI to install.

4. In the Step asking for XML File click Elipsis and select the correct settings file under EnvironmentSettings folder. Then click next.

5. BizTalk Application installed.!!

Silent Install / Unattended installation of BTDF using powershell:

https://randypaulo.wordpress.com/2012/01/31/automating-silent-install-biztalk-deployment-framework-btdf-using-powershell/

IIS 7.5 Missing Import Application – Deployment

When deploying a website  or a web service using Visual Studio 2010 we can use  a new deployment functionality called Build Deployment Package; this builds a zip file that you can import in IIS. The default location of zip is at ProjectFolder\obj\Debug\Package\.

 

However in IIS Manager, the default installation doesn’t include option to import our package, additional components need to be installed called Web Deploy (download). After installing the add-on you should be able to see the import option in IIS.