How do you handle the BizTalk deployment?

For the deployment of BizTalk applications we use a custom made tool within our company. We were curious to have some feedback from the community on the deployment of BizTalk applications. How do you handle the biztalk deployments at your company?

Our deployment manager tool is based on the idea to store all used objects in a database model (BizTalk artifacts, but also stuff like MSMQ, file locations, SQL objects, etc. …). Most of these objects are added using auto discovery of the BizTalk databases, so manually adding of objects is reduced to a minimum. Above all, BizTalk artifacts can have a different configuration (binding) defined per environment (test, dev, prod, …).
It also allows you to define all dependencies. Again most of these dependencies will be defined by the tool automatically.
This way of working makes it possible for the tool to define which actions need to be taken to deploy a certain application (or just a part of the application or only some objects). The deployment manager tool will define which objects need to be removed and redeployed (also unenlisting/disabling and starting/enabling artifacts will be done by itself). As a result, using the tool will allow us to deploy much faster, because the objects to redeploy are reduced to an absolute minimum and no complete redeploy is needed (like BizTalk Deployment framework does for example).

Another very useful and much used functionality is the possibility to define complete business/functional flows, including some generic components. This makes sure you can also deploy or redeploy a complete (new) flow (like an order flow for example) by itself, including all necessary objects (receive and send ports, file locations, etc. …).

The deployment can be done cross BizTalk applications. So the separation in applications is no longer deployment dependent.

The most important part is to set the database model correct and keep it this way… this will guarantee a much easier and faster deployment, where each environment has its own version of the current deployed objects.

 

Please answer following questions:

  1.      Is your BizTalk deployment automated (BizTalk deployment framework, custom scripting with MSBuild or BTSTask, powershell scripts, etc. …), or do you just do manual deployment using MSI and binding files?
  2.        Which deployment tools or scripts are you using, or have you used before? And what are your thoughts of these tools (benefits and complaints)?
  3.        What do you think of a tool like our custom deployment manager tool (using autodiscovery, etc. …)?

Thanks for your replies.

Advertisements

BAM deployment and partitioned tables

In this post I will give a brief overview of some problems that arise during deployment of BAM activity updates. In particular when partitioning is enabled for the activity the error below can occurre upon regeneration of the view that union the partitions.

“All queries combined using a UNION, INTERSECT or EXCEPT operator must have an equal number of expressions in their target lists”

 

When using BAM often the first deployment contains just a few data items and milestones. But when users become aware of the possibilities of this technology they want to build richer queries. Therefore I frequently need to add data items and milestones in my existing activity definitions. Of course business users don’t want to lose the existing BAM data. So I can almost never redeploy the activity definition which would delete all the data.
Now the bm.exe tool provides me with an option to update an activity. In this case no data would be lost. But after playing around for a while with this ‘update-all’ feature I discovered some problems. 
 
Let me explain what happens if one would like to add a data item to an activity which is already deployed and for which a view is created.
 
In my test environment I called my activity “BAMdeployment” and my view “BAMdeploymentView”. After first deployment the activity contains one data item StartData and two milestones Start and Stop. In the second deployment I want to add a StopData item to the activity and view. First I modified my spreadsheet and then exported the updated definition file.
 
Because I don’t want to lose data, I used the update-all command. Seems straightforward but I got the following error.

  

 
After looking around on the net I found some posts about this error and the solution is to first remove the view before running the update-all command. So after removing the view, the update-all completed without errors. The new data item StopData became visible in my view. Fine! But not for long.
 
The tests I did till now were done on a small set of data. But in a real scenario I probably have already a lot of instances of my activity in the BAMPrimaryImport database. Hence BAM archiving/partitioning becomes an issue.
 
After deploying an activity an SSIS package is created for partitioning and archiving the data. Here you can find the theory about this partitioning. Its all about performance.
 
When I tried to repeat the ‘update-all’ deployment strategy on my activity but now with partitioned tables I ran again into trouble. I got a new error.

 
 
Digging into this I ended up with a view in the BAMPrimaryImport database with the name bam_BAMdeployment_CompletedInstances. This view runs a UNION statement for every partitioned table. My custom view (the one defined in the spreadsheet) is afterwards build upon this one. This guarantee that all the data within my predefined activity window will be shown in the BAM portal.
 
Now the problem is that the partitioned tables are not updated with the new data item StopData. So the regeneration of the view bam_BAMdeployment_CompletedInstances fails. Only the first table bam_BAMdeployment_Completed has the new StopData column. While all the subsequent partitioning tables have not. 
 
To fix this problem all new data items and milestones must be added manually in the partitioning tables. While there will never be more than 254 partitioned tables with every activity, this is definitely an exhausting and error prone job to do. So I wrote a script to help you a little bit. Just fill in the name of the activity and the new data item or milestone you want to add. Be sure that you use the same names as given in the definitionfile of the activity. The script will update every partitioning table. 
 

All at once, the right order of actions to take

1/ Update the activity in the spreadsheet and export the definitionfile.

2/ When partitioned tables exist use the script to add the new items.

3/ Remove the views defined for the activity.

4/ Execute the bm update-all command.


-- Script to update the partition tables after adding a data item or milestone to the activity definition.

DECLARE    @activityName nvarchar(128)
DECLARE    @column nvarchar(100)
DECLARE    @columnType nvarchar(100)
DECLARE    @instancePartition sysname

SET @activityName        = 'BAMdeployment'
SET @column             = 'StopData'
SET @columnType         = 'varchar(50)'
BEGIN

IF NOT EXISTS(SELECT ActivityName FROM bam_Metadata_Activities
                 WHERE ActivityName = @activityName)
BEGIN
RAISERROR (N'ActivityDoesNotExist', 16, 1)
RETURN
END
   
DECLARE partition_cursor CURSOR LOCAL FOR
SELECT InstancesTable
FROM [dbo].[bam_Metadata_Partitions]
WHERE ActivityName = @activityName
ORDER BY CreationTime ASC

OPEN partition_cursor
FETCH NEXT FROM partition_cursor
INTO @instancePartition
WHILE @@fetch_status = 0
BEGIN

EXEC ('ALTER TABLE ' + @instancePartition + ' ADD ' + @column + ' ' + @columnType)
    
FETCH NEXT FROM partition_cursor
INTO @instancePartition
END
END
GO

BAM deployment in BizTalk Server 2006

Personally I found that BAM is one of the most useful features in Microsoft BizTalk Server 2006. It brings down the gap between business and IT. Unfortunally like most BizTalk architects/developers I struggled with deployment on my production environment. If you don’t need to keep the data for historical reasons or analysis then deployment of a BAM flow is easy and straight forward. You can just delete your flow and recreate it again. In case your customer requires you to keep your data then deployment becomes trickier.

For those scenario’s our team created the following check list:

Type Questions Undeploy Remove view Update Deploy Data loss
Activity Add Activity    

1

 

N

Activity Add Business Data field    

1

 

N

Activity Add Business Milestone field    

1

 

N

Activity Modify data type of Business Data field (a)

1

   

2

Y

Activity Delete Business Milestone field

1

   

2

Y

Activity Delete Business Data field

1

   

2

Y

Activity Delete Activity

1

     

Y

View Add View    

1

 

N

View Add Activity  

1

2

 

N

View Add Business Milestone item  

1

2

 

N

View Add Business Data item  

1

2

 

N

View Add Alias    

1

 

N

View Add Measurement  

1

2

 

N

View Add Duration  

1

2

 

N

View Add Group  

1

2

 

N

View Add Dimension  

1

2

 

N

View Modify Alias  

1

2

 

N

View Modify Duration  

1

2

 

N

View Modify Group (b)

1

   

2

Y

View Modify Dimension  

1

2

 

N

View Modify Measurement  

1

2

 

N

View Remove Group  

1

2

 

N

View Remove Duration  

1

2

 

N

View Remove Alias  

1

2

 

N

View Remove Business Milestone item  

1

2

 

N

View Remove Business Data item  

1

2

 

N

View Remove Dimension  

1

2

 

N

View Remove Measurement  

1

2

 

N

View Remove Activity  

1

2

 

N

View Remove View  

1

2

 

N

Pivot Add Pivot Table (c)  

1

2

 

N

Pivot Modify Pivot Table (c)  

1

2

 

N

Pivot Remove Pivot Table

1

   

2

Y

 

  1. Change data type of a Business Data field is not possible. The field needs to be dropped and recreated again.
  2. Change of a group is not possible. The group needs to be deleted and recreated
  3. Update will not give an error but nothing will be changed

     

The headers of the check list reference the different actions that one can do with the bm.exe tool.

Some examples:

Remove Business Data field from activity

In this example there are 2 definition files:

  • SampleBAMDefinitionFile V1.1.1.1.xml: contains the current deployed BAM activity
  • SampleBAMDefinitionFile V1.1.1.2.xml: contains the new BAM activity without the Business Data field

In order to update BAM the following commands should be executed:

bm.exe remove-all -DefinitionFile:”SampleBAMDefinitionFile V1.1.1.1.xml”
bm.exe deploy-all -DefinitionFile:”SampleBAMDefinitionFile V1.1.1.2.xml”

 

Remove Business Milestone item

In this example there are 2 definition files:

  • SampleBAMDefinitionFile V1.1.1.1.xml: contains the current deployed BAM activity
  • SampleBAMDefinitionFile V1.1.1.2.xml: contains the new BAM activity without the Business Data field

     

In order to remove the Business Milestone from the view only the following commands should be executed:

bm.exe remove-view –Name:SampleBAMView
bm.exe update-all -DefinitionFile:”SampleBAMDefinitionFile V1.1.1.2.xml”

In case the Business Milestone needs to be removed from the activity then the above logic will not work. In this case the following commands should be executed:

bm.exe remove-all -DefinitionFile:”SampleBAMDefinitionFile V1.1.1.1.xml”
bm.exe deploy-all -DefinitionFile:”SampleBAMDefinitionFile V1.1.1.2.xml”