Microsoft Integration MVP

I am so pleased to announce the new MVP Category.

Microsoft Integration MVP

I have always been doing integration, for the last 12 years I've been doing integration, it's just something about being able to reach out to other business systems and not just make them talk to each other make the connections between them dance. Lay down the right architecture and make it work.

The technology I used also varied, be it WCF, BizTalk, Azure, SQL Azure, Azure XYZ. , SharePoint, SQL Server or anything else, when you do integration you previously had to choose which MVP to be…

Now thanks to Microsoft you don’t need to choose anymore…. 

I could not be happier with the name as it correctly reflects exactly what we do, I will write posts and articles on any of these topics all under the banner of integration, and all on the Microsoft technology stack. Bring it on I say.

The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the distributed transaction.

We had a WCF-SQL send port that was to insert a large data set into the database, and then execute a stored procedure when done, this can all be sent as one message to sql and it works it out.

The problem is that this took a very long time, we increased time outs but were still getting a time out every time, with the error: 

System.Data.SqlClient.SqlException: The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the distributed transaction.

Server stack trace:
   at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.EndRequest(IAsyncResult result)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at System.ServiceModel.Channels.IRequestChannel.EndRequest(IAsyncResult result)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.RequestCallback(IAsyncResult result)

We looked around and found that the timeout for the msdtc controller was in the machine,config.

%Windows%\Microsoft.NET\Framework\v2.0.50727\CONFIG\machine.config

Or if you run 64bit.

%Windows%\Microsoft.NET\Framework64\v2.0.50727\CONFIG\machine.config

<system.transactions>

<machineSettings maxTimeout=”01:10:00″ />
</system.transactions>

You may find this already in there, we had 00:30:00 as the timeout, we increased this, and it worked.

Problem solved.

To Integrate or to Integrate Badly

When someone chooses to integrate there is always the wrong way and the right way to go about this. Just because you chose to integrate does not mean you did a fantastic job about it. 

One such example I came across recently was a nameless organisation decided to integrate their systems, they provided a very good level of integration and it was all working fine, then they got over zealous, a contractor they hired thought hey I can do integration also, the company not knowing any wiser said OK sure let’s do it. 

The integration was a simple one, I’d like to change my surname, and it should update all the systems, because I am married now and have taken my husband’s name by choice. 

The old way of doing this was very manual and very cumbersome and took a long time to implement.

We already provided integration of other user profile details, which were updated in the various systems in the organisation. The contractor thought hey I can do it too… 

The surname request update had a fancy front end, when the user submitted this, their request was sent as an email to someone in HR, the first problem. 
 
They would manually make the changes required in the system… The second problem.

What is worse, is that the email got it wrong, it said update my first name to this name, and not my surname, so the person in HR went and updated their first name to their new surname…  This person was outsourced from the organisation. The third problem.

This had flow on effects, that had not been thought of, when someone changes their name, their AD name needs to update, their email address needs to change, and a whole bunch of other systems need to be informed. 

To get around this they just deleted the user and made a new one as if they had just joined the organisation, the flow on was very very bad… Major problem goes with out saying.

All of their access gone, all of their email gone, their permissions to all the internal systems gone, this would have happened even if they had changed their surname. 

The loss of productivity for this person was about two weeks’ worth, the person was not a low level admin, they were senior and on a good rate. 

Take this against the decision to use a contractor who said yeah we can do it,  probably $1000 for the efforts, then add the down time of the one employee, $20000 in loss of productivity and in ability to work, you are not only paying for them to work but they can’t work and produce income for the organisation. 

Now multiply this by a low estimate of 5 people who may have been effected by this across an organisation of 7000 people…  

You have a net loss to the organisation of: $100,000  plus your low cost of implementing this in the first place.  

Take this and compare against doing it correctly in the first place, taking the time to understand the impact, going and updating the various systems, in near real time, and ensuring that the user was good to go within 15 minutes. 

A cost estimate for this would be somewhere between  $20,000 to $60,000. Had more than 5 people requested this change, think of the cost to the organisation.

It becomes very clear that you need to integrate, and you need to do it properly. You need to have the right people working with you who understand this and are able to see these issues well before they happen. 

Never believe someone who says yeah we can do that, no problem, it is easy, we will just email someone, or we will add a manual step, or we will just leverage the existing process or method, or we will just delete them and put them back… these things make me cringe… don’t do it… ever! 

Talk to the professionals, they have done it all before. 

Business Process and Integration

The corporate landscape of today includes a wide variety of heterogeneous business applications.

Each designed to solve a specific and specialised business problem. These applications are often not designed to work together seamlessly. This creates a major challenge when information needs to be extracted or combined for various purposes across the enterprise. The question then arises, “How do organisations extract relevant information and deliver it in the right format, to a destination either inside or outside of the organisation, across functional domains, at the right time?” This business challenge makes Business Integration a high priority.

The number one organisation I recommend to solve this problem is Connexion Technology I am proud to announce a partnership between myself and Connexion Technology, one that will see this organisation move to the forefront of this space, and help countless organisations.

Accessing the SharePoint user profile service from BizTalk

I have a client scenario where weird 3^rd party system 1, sends me an update of some information, from time to time, about a user, this information needs to be instantly replicated into SharePoint.

I could use BCS but it’s not instant…  It’s coming to me via BizTalk, so I have it in near real time, so 

I can update SharePoint can’t I??… well yes it seems you can…

There is the web service that SharePoint quite nicely exposes:

http://%7bserver%7d/_vti_bin/userprofileservice.asmx

It has many methods one of these is: ModifyUserPropertyByAccountName

It’s a one way send in BizTalk. 

When you add a reference, you need to add a generated item, that consumes a WCF Service.

You will get 2 schema s an orchestration and a port binding, the schema s are useful, the orchestration you can choose to use it or not, it contains a bunch of multi-part message types,  and a massive port for every single method.

The port is useful, because it contains the operations, you will need to use when you create the port, so one port can have many operations to the same web service, they give you the binding for this port, which is very nice, I suggest you use it.

Here is where all the nice stuff ends, and the really interesting stuff begins. If you use this, it does not work. 

You may get:

Error details: System.ServiceModel.FaultException: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>Server was unable to process request. ---&gt; Invalid String Value: Input must match string data type.</faultstring><detail /></soap:Fault></soap:Body></soap:Envelope>

This made no sense to me, as I was sending a string, and updating a string…..

If you look at the schema for the message:

value is defined as xs:anyType, which means you can put anything in here, which is correct, I want to, a string, a datetime, a int. So the schema is flexible…

The wsdl is very vague:

      <s:complexType name="ValueData">

        <s:sequence>

          <s:element minOccurs="0" maxOccurs="1" name="Value" />

        </s:sequence>

      </s:complexType>

The value is does not specify xs:anytype however because it does not specify a type BizTalk interprets this, bad sharepoint.

However it does not go far enough… and hence it is a real let down when you have gotten past the authentication issue to find this. See my other post to fix this.

If you call the method from .net code, it works… however to find out why it works, you look at the xml that this call generates:

<ValueData>

   <Value xsi:type="xsd:dateTime">2012-06-25T10:01:17.486123+10:00</Value>   </ValueData>

The “value”  has an uppercase V, BizTalk puts is with a lower case V.  so that’s the first fix.

The next problem: xsi:type ??

It’s not even in the schema, it’s not in the wsdl.  I try and try and try to get it in to the schema, it’s not going to happen.

I managed to add an attribute to the value, called type, however it’s coming up as:

<ns0:Value ns0:type

Now the values are being set in SharePoint, all to NULL because it cannot interpret the type.

Now I know what I need to make the message look like, I have BizTalk pipelines… I can touch up the message before I send it to SharePoint.

So I go and touch up the message and the thing works perfectly. !!

I can now communicate from BizTalk to SharePoint, to update the user profiles.

My port looked like this:

Accessing an authenticated web service using BizTalk and impersonation

I need to access the SharePoint user profile service.

SharePoint quite nicely exposes:  http://{Server}/_vti_bin/userprofileservice.asmx

It is authenticated, quite rightly so, however the user I am accessing it with has permissions, however it still asks for username and password.

I was getting this error:

The adapter failed to transmit message going to send port "WcfSendPort_UserProfileServiceSoapOneWay" with URL "http://{Server}/_vti_bin/userprofileservice.asmx". It will be retransmitted after the retry interval specified for this Send Port. Details:"System.ServiceModel.FaultException: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>Server was unable to process request. ---&gt; System.ServiceModel.EndpointNotFoundException</faultstring><detail /></soap:Fault></soap:Body></soap:Envelope>

Server was unable to process request. System.ServiceModel.EndpointNotFoundException

Not much help to me…

After some searching I found I needed to use impersonation.

The main reason was from .net I can simply do:

WSService.Credentials = System.Net.CredentialCache.DefaultCredentials;

Now the current user is impersonated and if they have permissions it works. GREAT.

I’m in BizTalk, and I’m on the WCF port…

The port is using wcf-basichttp as the protocol, and it should be able to just I have no idea how.

Well the solution I found is rather simple.

Use a wcf-custom port.  

Setup the basic-httpbinding.

First of all set the transport to client credential type ntlm 

 

Then flip over to Behavior

It will be blank.  Right click end point behavior: 

Select add extension.

Add client credentials and configure like so:

 After doing this my error went away.

Publishing an InfoPath form to Sharepoint 2010 for WebBrowser

Publishing an InfoPath form to SharePoint 2010 is not that difficult, what is difficult is getting it published when it has code behind for the use in the Web Browser.

I recently battled with this, and I must say it’s better than it used to be, making InfoPath VERY powerful, particularly if you can use code behind, and even call your own .net dll’s from the form.

For most they have got somewhere, but have not got it fully working.. if you are one of these people read on. 

Do you want to activate it in the site collection, but can't upload it?

First you need to do this:

1. Open SharePoint 2010 Central Administration.
2. Click General Application Settings.
3. On the General Application Settings page under InfoPath Forms Services, click Upload form template.

You GET: ERROR: This form template has not been published. Open the form in the InfoPath Designer and publish the form to SharePoint using the Administrator-approved form template method in the Publishing Wizard.

Do the following:

Important: Ensure that the security and trust is set to full trust, with a certificate.

1. Publish the form to SharePoint from InfoPath.

2. Note down where you published it: http://msite/template.xsn

3. Go to your web browser and go to this URL, download the PUBLISHED template.xsn

4. SAVE IT locally.

OR

4a. Better way: Publish it to a network location say c:\MyFormTemplates, you do not need it in SharePoint YET.

5. Now upload the {template}.xsn, (Rename if you like) to SharePoint admin.

You should get: The form template has been successfully uploaded to the farm. To make the form template available in a site collection, activate the form template from the Manage Form Templates page or from the feature activation page in the site collection.

6. Now you can activate it in your site collection... etc...

Go to your site where you want to use the form.

Site settings / site collection administration

Site collection features

Look for your Infopath Form Template, (it's sorted alphabetically)

Press Activate

---

THE KEY to enabling the form for the browser is in how you set up your form library...

Under Site/{FormLibraryName}/form library settings/ advanced settings.

SAY Allow management of content types = YES

Then hit ok.

Now go down to content types, and defined your content type, say add from existing site content types, which you just activated... You do remember the NAME of the form you uploaded in admin before??

Make sure that it is the default content type, and everything will be good. 

Grant permissions to your users, and have them press new document, and up comes your form in the browser...

You can then put your workflow on the form library, or hook BizTalk up to the form Library, and you have full end to end Form and Workflow.