Connection to Microsoft Exchange is unavailable on Lync 2010 Phone Edition

You know, I’ve learned something today. When something does not work as expected, there is a reason for that, even when you don’t see it… right away.

One of our customers had an issue with Lync Phone Edition calendar integration. The phones simply refused to connect to EWS, period. To make the issue more puzzling, a phone signing form Public Internet worked, but one on LAN did not. Customer uses BIG-IP F5 hardware load balancer as HLB appliance.

The troubleshooting process started with reviewing the phone logs (.CLG2 in particular). I know, it is very hard to read them logs, but TextPad did the job. I was looking the calls for Autodiscover web services. LPE uses Autodiscover to obtain both internal and external EWS URLs. In the log I found the following: “NAutoDiscover::DnsAutodiscoverTask::TryAutodiscoverUrls: Exception with this url. hr=0x80072f0d”. This error translates to “The security certificate on the server is invalid”. What??? VeriSign Certificate is not trusted??? Tens of thousands of users connect to their Exchange accounts from desktop and mobile devices in and out, and yet LPE does not like the certificate.

There are three things I need to mention here:

1. The customer uses certificate signed from Public CA on the CAS array
2. Because of this fact, the customer already added VeriSign Root CA to the Lync certificate store by following the procedure described in Kevin’s blog: http://ocsguy.com/2012/05/19/lync-phone-edition-connection-to-microsoft-exchange-is-unavailable
3. Customer set “internal” and “external” VIPs on F5 for all Exchange services

First step was to examine the certificates deployed to CAS array. This is very important in order to establish the number of intermediate certificates. In the first example we have one intermediate – VeriSign Class 3 Secure Server CA.

Now, let’s look another certificate:

In this case we have two intermediates – the issuer (VeriSign Class 3 Secure Server CA – G3) which is sub for VeriSign Class 3 Secure Server CA.

I was puzzled as of why web browser did not fail when visiting autodiscover.domain.com, but the phone did, until realized - browsers are capable of following the chain all the way up to the Root CA i.e. F5 presents Identity Certificate – server.domain.com and the browser will follow the chain up to the Root CA.

There are many, many posts in Internet about mobile devices failing to connect to EWS. Additional research shows that mobile devices, due to the limited size of the Certificate Store and platform specifics, require ALL INTERMIDIATE certificates to be presented along with the identity certificate in the initial handshake. This way the device will follow that chain “internally”, until the Root CA in the local store validate the chain.

We already know the limitation of Aries platform in this regard, and so I thought – what if we have the same limitation and behavior as Mobile Devices have? This prompted close examination of the handshake between client and server. Below is an example of successful handshake:

What is important here - we have immediate flow of type “Application Data”, which means the handshake was completed successfully, a trust was established, and data exchange has begun.

Let’s now look closely what happens:

After initial Client and Server “Hello”, the server will offer the Certificate(s).

As we continue to drill, we see that two certificates were offered

…server identity and the intermediate issuer, which matches the chain as we saw already:

All Lync Phone Edition have to do now is to check if “VeriSign Class 3 Secure Server CA” can be trusted, and we do because the Root – “VeriSign Class 3 Public Primary CA” is in the Trusted CA Root Store of LPE.

Now let’s see what was happening in the customer environment… we would expect to see this same flow internally…

During the Certificate presentation, we note that the ONLY certificate offered from F5 is the “Identity Certificate” and no Intermediate was present:

Of course, LPE could not follow the chain and the handshake failed miserably. We asked for autodiscover.dmain.com and received said Server Identity Certificate, but could not validate the Issuer - VeriSign Class 3 Secure Server CA.

We spoke with the F5 team, and they confirmed that the Internal VIP for CAS array had only Server Identity Certificate assigned, but not intermediate. After assigning the correct Intermediate Certificate to the internal services for Autodiscover and EWS, the issue was resolved.  I am still baffled from the fact one can/must assign intermediate certificate to service on F5. This should be done automatically during the certificate import…

***This concept or troubleshooting can be applied when investigating connectivity issues between CAS and Mobile Devices where not only F5, but other HLBs are used and we have problems with SSL.

Lync 2010 - the Dial Plan mystery

Enterprise Voice still reminds one of the mysteries of Lync deployment. While IM&P and conferencing is relatively well known and documented, EV is sort of gray area due to the fact it involves more than server and roles installation. The interaction with PSTN (Public Switching Telephone Network) often adds complexity that throws Lync Administrator in uncharted territory…

In this post, I will go over the best practices of creating Dial Plan. I have seen many people, overexcited of the possibility to make a phone call via Lync, often would omit the proper configuration and use “one fits all” rules.

Lync uses E.164 number format i.e. “international public telecommunication numbering plan”. As defined per ITU-T recommendations, E.164 format can have maximum of 15 numbers and must be prefixed with “+” sign. An example of such number is +14785550000 where “1” is the country code (US), “478” is area code (Central Georgia), “555”  is exchange (local provider’s PBX) and “0000” – extension

A very simplified explanation of the routing logic Lync uses is this: when a number is dialed from Lync client, the first action is to look if this number can be matched internally – to a user or any other object that have E.164 number assigned. If “Yes”, the call is routed internally to the object, if “No”, if appropriate action is defined, the call is “dispatched” to the destination. If there is no “routing definition”, the call fails. It is so simple!

Back to Lync and E.164. For EV enabled users, we must define the number in E.164 format, like +14785551234. The $64,000 question is – how in the world I dial “+” from my phone? From my Polycom CX600, I can dial 14785551234, but then this number will not match my user’s number because it does not have ”+” on front. Here comes the mystery of “normalization”. While “normalization” is the correct word (since we bring ANY number dialed to a “normal”, or E.164 format), I noted that people are handling the term “transformation” better. In any case, to “normalize” 14785551234 to +14785551234 OR to “transform” it - same action, just worded differently.

Here comes the magic of “Dial Plan” in Lync. Dial Plan uses RegEx (Regular Expressions) to evaluate the string and transform it (if match is found). Simply said – comes down to “what I am dialing” and “what final result I want to achieve”.

^(\d{11})$

+$1

The above can invoke suicidal thoughts in any man or a woman! What does it mean??? Will I ever be able to understand how this works?

Now let’s use this approach – “when I dial something, my goal is the end result to be something else”. In this case, “When I dial 14785551234, I want the final result to be +14785551234”. The above RegEx will do just that. To write RegEx freely, however, requires you to be at least member of Mensa society or above. Fortunately, Lync Control Panel provides graphical representation of “when I dial something, my goal is the end result to be something else”.

It cannot be any easier:

Starting digits – it does not matte

Length – exactly 11 digits. (this means that RexEx will “count’ the number if digits and will not process if it is different from “11”

Digits to remove – in this case “do not remove digits”

Digits to add – add “+” on front.

Let see what happens when we test our RegEx:

We just normalized (or transformed if you wish), 14785551234 to +14785551234.

Now that we know what Normalization (or transformation) does and how to use it, let’s pull back and think about our environment. Every Lync deployment is like DNA – there could be a partial match (if two companies in same town, using the same PSTN provider deploy Lync Enterprise Voice) and yet it will never be the same because of company specifics. Proper evaluation and planning is absolutely vital for successful outcome of any EV deployment.

Here is an example if such process.

1. In Milledgeville we dial 911 for emergency
2. Our current “internal extensions” are 4 digit long
3. We dial local numbers by dialing 7 digits only.
4. Out of state we dial 11 digits (1 + area code + 7 digits)
5. International calls – any length (011 + country code + number)

Now I will use Lync Control Panel to “convert” this to Lync Dial Plan.

***I will create new User Dial plan. For differences between Pool, Site and User dial plan, refer to the product documentation.

Here comes my first Normalization rule – for 911.

  Next is handling of 4 digits internal extensions.

***This example requires explanation. All numbers in my organization are from the block 1478387xxxx. From this prospective, it is safe to assume that when user dials (or “mash”, as they call it on South) four digits, the intention is to reach a colleague from the same organization and so, the number will always start with 1478387. Here is an illustration:

Now - local calls

***Because those calls are “Local”, they always stay within the local Area Code (478). We add +1478 on front of the number (to become E.164).

Next – Long Distance calls. There are two ways to dial LD – 10 and 11 digits. I will create two rules to capture both cases.

Lastly, a rule for International calls.

Just for fun, I will throw one more example. The Exchange Auto Attendant’s E.164 number in my organization is +14785559001. I want to provide an easy way for my users to dial the AA and reach a colleague by speaking the name. I selected *99 as “quick dial”.

The test result:

Here is the final result of my hard work:

At the end, I will commit the changes i.e. make it permanent.

In the next post, we will look at Lync Voice Policy and how it interacts in our EV environment.

Lync 2010 on prem - Cloud UM integration

“Cloud computing is like sex in High School. Everybody is talking about it, only few did it, and they didn’t know what they are doing.”

                                                                                          Found somewhere in Internet

I have been thinking recently of one hypothetical scenario – let’s say my company has domain “lynclog.com” and we host Exchange and Lync on prem. We acquire small company B with domain “myuc.us”. The management decides to decommission Exchange Server in Company B and move all users to Office 365 Exchange services. However, because "lynclog.com" already utilizes Lync Enterprise Voice, we also want to decommission Company B’s PBX and provide EV to” myuc.us” users. The main question is - how to resolve the problem with Unified Messaging for Company B?

Let’s see if this is possible.

First, of course, I must sign up for Office 365 service. I created tenant “forblog.onmicrosoft.com”, added “myuc.us” and verified it as outlined on the online documentation.

For the purpose of this post, I created user “Myuc User” with ID mu1@myuc.us. This represents all users from Company B and the assumption here is the migration to Exchange Online has been competed, mailboxes migrated etc.

Next step is to prepare the Cloud service for Unified Messaging. I will create UM Dial Plan:

…and we can see, a default policy was automatically created. I might modify it later.

***I have not created Auto Attendant at this point – I am more interested if this concept will work to begin with…

Back in Users and Groups, I will enable our user for Unified Messaging

This concludes my Cloud based user configuration. now let's prepare hosted Lync for Cloud Integration.

Run the following command from Lync Management Shell:

New-CsHostingProvider -Identity "Hosted UM" -Enabled $True -enabledSharedAddressspace $True -HostsOCSUsers $false -ProxyFqdn "exap.um.outlook.com" -IsLocal $False -VerificationLevel UseSourceVerification

Get-CsHostingProvider -Identity "Hosted UM" cmdlet should return the following:

***I hit a bump at this point. Took me for a while to figure that the routing must be done toward the authoritative cloud domain. To see which domain is authoritative for my setup, I had to use Remote Power Shell:

$cred = Get-Credential

$s = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell -Credential $cred -Authentication Basic -AllowRedirection

$importresults = Import-PSSession $s

...and verify if our session is OK

 Get-Mailbox

 When i ran: Get-AcceptedDomain , the output showed that only "forblog.onmicrosoft.com" is Authoritative and I will use it in my Lync configuration.

Next, create new Hosted Voice Mail Policy:

New-CsHostedVoicemailPolicy -Identity "MYUC Domain" -Destination exap.um.outlook.com -Description "Hosted voicemail policy for myuc.us domain." -Organization "forblog.onmicrosoft.com"

***Note that I am using  "forblog.onmicrosoft.com" as Organization, because this is the only Authoritative domain returned.

Get-CsHostedVoicemailPolicy -Identity "MYUC Domain" should return

***Yesterday I already added "myuc.us" as "Additional Supported Sip Domain" in my Lync topology, made the necessary changes in the public DNS, reissued the public certificate to include the new domain, you know the drill..Also, mu1@myus.us was added to Lync and provisioned for Enterprise Voice.

Now I will set HostedVoiceMailPolicy for this user to True. This way, when voicemail is to be deposited, my Lync on prem will route the call to the Cloud.

Set-CsUser -Identity lynclog\mu1 -HostedVoiceMail $True

...and grant the policy I created earlier:

Grant-CsHostedVoicemailPolicy -Identity lynclog\mu1 -PolicyName "MYUC Domain"

Let's examine our user' properties now:

Time to test our setup.

The call was connected and I left new Voice Mail.

***One last note - in this test,on my Edge Server, I used Public Certificate issued form StartSSL.org. Office 365 did not have any problem with it...

Lync Mobility Deployment - Part I

The big day has come. I will install and configure Lync Mobility Support in my lab. When done, I should expect my environment to have three new services:

• Mobility service,  new virtual (Mcx) directories under both Internal and External websites of my Front End Server(s). This service will support various mobile devices and provide IM, presence, contacts etc.
• Autodiscover service – new virtual (Autodiscover) directories under both Internal and External websites of my Front End Server(s). This service will be used from mobile device to locate the internal and external services which supports mobility.
• Push Notification Service – This actually is a cloud-based service located in Lync Online datacenter. Because devices based on Windows Mobile and Apple iOS are designed to go in “suspended state” when not active, those cannot actively respond to new events (while suspended) and notification to “wake up” the application must be supplied somehow. This process will be described in later post.

*****I would like to emphasize on the importance of utilizing Autodiscover service. As we will see later, using Autodicover is not mandatory and users can manually specify the mobility service URL’s (internal and external). While this is relatively simple task, we as Administrators or support team members should be prepared for increased number of support calls (simple typo while entering the service URL will cause the signup process to fail) shall decide not to deploy Autodiscover service.

With this in mind, let’s proceed with introducing Lync Mobile device support to our environment.

First, I will need to make the necessary DNS entries. I already decided to use Automatic Discovery and for that, I will make an entry in my Internal DNS for ‘lyncdiscoverinternal.lynclog.com pointing to the IP address of my only Front End server. Now, I do not use Director Role in my lab, else I would ponit to either the Director pool.

***I could also use CNAME record pointing to the FQDN of my FE server (or Director as described above), but this would mean an extra hop since after obtaining the FQDN, a second call to the DNS server will be made to discover the IP address of the server. Both records are valid scenario and it is up to you to decide which model to use.

Next, I will make an entry in my External DNS (currently hosted with GoDaddy and will not be shown here). I will point this record to the IP address of my Reverse Proxy, where new listener will be created on later stage.

Moving along on with the prerequisites. The new web service requires “Dynamic Content Compression” role service on our Web Server. This can be done either via Add/Remove roles or Lync Management shell by issuing the following commands:

Import-Module ServerManager

Add-WindowsFeature Web-Server, Web-Dyn-Compression

***If you are using IIS7.0, refer to the product manual for additional configuration steps.

***Before proceed further, it is imperative that you already installed “Cumulative Update for Lync Server 2010: November 2011” a.k.a. Lync CU4. Once installed, Lync server will be extended with new features.

In order to install the Mobile Support bits, I must set the ports for the Internal and External Web Services via Lync Management Shell:

Set-CsWebServer –Identity FQDN –McxSipPrimaryListeningPort 5086

Set-CsWebServer –Identity FQDN –McxSipExternalListeningPort 5087

…and run Enable-CsTopology –verbose to make the change.

***Note that this must be done on each Front End and Director in your environment where Mobile devices will be supported.

Navigate to "C:\ProgramData\Microsoft\Lync Server\Deployment\cache\4.0.7577.0\setup" and drop McxStandalone.msi there.

In Lync Management Shell, navigate to "C:\Program Files\Microsoft Lync Server 2010\Deployment" and run Bootstrapper.exe. At this point, we will see the two new web services installed.

 

So, I now have the virtual directories but what about the certificates? Indeed, I need to request and assign new certificates from my Internal CA to include the new DNS names I added above. Fortunately, with CU4, Lync Deployment Wizard become "smarter" and will do this for me.

Now I can test the deployment as described in the Deployment guide:

So far, so good. Later today I will deploy new listener on my TMG server and meanwhile hope the clients will be published soon...

Lync Mobile devices support

The greatly anticipated support for mobile devices has arrived (bits and deployment guide) and… I sense great disturbance in the force already. “Voice and video over Wi-Fi Microsoft did not provide?!? How embarrassing… how embarrassing”. I did too fall to the dark side for about… 30 seconds.

We are professionals. We travel all over the country. We spend time on airports, in hotel rooms, and other places where Wi-Fi access is provided. Let me ask as professional – How satisfied are you with the quality of service? How often you had to turn off the wireless capabilities on your phone to get your email over 3G or 4G? What business value an application that “works over Wi-Fi”, but actually prevents you from doing business has? Who would be to blame for that missed call i.e. missed business opportunity?

Fortunately, we have an awesome feature called “Call via Work”. The way it works is - our mobile application instructs Lync server to dial the destination number, then dials our device (could be mobile number or a landline) and bridges the call. The number presented to the called party is our Tel-Uri i.e. Business phone. Why is this important? First - I don’t want to reveal my cellphone number to everyone I call. Second - because the Caller ID shown to the called party will be my office phone number, returned calls will terminate in my office. From my employer’s prospective, if I (an employee) leave the company, my former number will be assigned to my successor and calls for me (my position, that is) will be answered from the new person.

And, of course, there is something else we do not talk about, but must be taken in to consideration. If you are reading this post – you are Lync Administrator or an user the least. You or your organization has selected Lync as Unified Communications platform because of the business value and the quality of system as a whole. Now, if adding “perks” were to require rewriting of parts or entire modules of such complex system, are you willing to take the risk of jeopardize  your business operations? There must be a reason why voice and video over Wi-Fi was omitted in this release and the sooner Microsoft come forward and explain those reasons, the less any “disturbance in the force” will keep us away from what we should do in first place – stop bitching, grab the manual and make sure we understand the new functionality of Lync Mobile Support as it is now.

Lync 2010 CU4 is out

Hurray, the long anticipated Cumulative Update is out. You can download it here:  http://www.microsoft.com/download/en/details.aspx?id=11551 Of course, I would recommend to use LyncServerUpdateInstaller.exe, unless you have special reason to omit some role update.

Description of the fixes can be found here: http://support.microsoft.com/?kbid=2493736

Now, was I disappointed Mobility was not included in this package – not really. I do believe, however, that Cumulative Update 4 introduces the changes necessary to support mobility. I am facing the fact Mobility is not yet available philosophically – “It will be ready when it is ready”.

Lync is now an enterprise grade system and we should no longer expect something that will require “fixing” right away. Like I always said “Patience is a virtue” ...especially when comes to dial tone...