Wednesday, December 11, 2013

User might experince delay when join Lync Online Meeting.

Recently I’ve worked with large enterprise customer on troubleshooting strange issue with Lync Meeting join. Users from the pilot (50 people) group would report prolonged (up to 30 seconds) delay between the splash screen and the actual Lync Meeting join. A pattern was not observed (although there is one, just not obvious at first) – it would happen to random users, at random time, joining different meeting ID’s on different pools.

In this article I will explain the background of the “issue” and describe possible solution.

When user click Lync meeting link in format https://meet.contoso.com/user/M33T1NG1D , the request is processed by Director Pool member (if Simple URL points to Director), where after Meeting Discovery, Director Web Service will respond with “301 Moved permanently” to the original request. Client then sends new GET request with destination Web Service of the pool where the meeting will be hosted and meeting join completes.

As Lync server is installed on Windows Server, HTTP requests are served by Internet Information Services (IIS). The web sites (Internal or External) and Virtual Directories within the sites are driven by IIS Worker Process and are logically defined within IIS Application Pools.



As we see, not only whole Web sites (Internal or External), but also Virtual Directory within a web site might be configured to run under dedicated Application Pool.

One aspect of IIS Application Pool is “Recycling”. This process is defined in TechNet as “Internet Information Services (IIS) application pools can be periodically recycled to avoid unstable states that can lead to application crashes, hangs, or memory leaks.” When we install Lync server components, a default value is configured - 1740 minutes (29 hours).




When the Application Pool recycles, all items in the memory are released, a new w3wp process is created, configuration read and applied, .NET reloaded and we site is back serving requests. Under normal circumstances, had we had a request for simple web page as https://dialin.contoso.com , user would not experience visible delay because there is no complexity in assembling and serving this page.

However, Meeting Join request requires extensive server side processing, not discussed in this article. In reality, it takes about 22 seconds for the very first meeting join request to be served after Application Pool Recycle. This fact might cause inconvenience under certain circumstances – in my case a very large front end pool (nine pool members) and only 50 pilot users on this pool. Because the pool web services are Hardware Load Balanced, there is great chance that unique user would hit pool member where the Application Pool was just recycled and expertise 22+ seconds delay. Reports of type “It took forever to join the meeting this morning” were pouring in and frustration was growing fast.

The solution to eliminate the delay is rather simple – monitor the System Event Log for EventID 5074, examine the event description for which Application Pool was just recycled, and fire up Scheduled Task to execute PS script which will simulate real meeting join request. The request will perform “Application Pool warm-up” and because we do so immediately after recycle even, chances a real life user to attempt to join after Pool Recycling, but prior to Scheduled Task execution is practically non-existent.



Steps to configure Application Pool meeting join warm-up:
  

Step 1: Create folder “Scripts” on C:\
  • Inside this folder, create text file MeetExtWarUp.txt
  • The file should contain single line Invoke-WebRequest https://FE_Server_FQDN:4443/Meet?key=A1B2C3D4 where FE_Server_FQDN is the FQDN of your SE or EE Lync front end server.
  • Rename the file to MeetExtWarUp.ps1
  • Follow the steps above to create second .ps1 file where the text is 
    Invoke-WebRequest https://FE_Server_FQDN/Meet?key=A1B2C3D4
***Note that we use “:4443” for the External Web Site. It will be omitted in the file used for the Internal Web Site.


Step 2: Create Scheduled Task to execute the .ps1 script when AppPool recycle event is detected.

  • Start Task Manager as Administrator
  • Expand Microsoft/Windows node, right click and select "Create Task"



  • Name the task (WarmInternalAppPool in this example)
  • Check "Run with highest privileges"
  • Click "Change User or Group" button and select SYSTEM, then click OK.




  • Select "Triggers" tab.
  • Click "New"
  • In the "New Trigger" window, select "On an event" from "Begin the task:"
  • Toggle "Custom" under Settings
  • Click "New Event Filter" button



  • In New Event Filter window, click XML tab.
  • Check "Edit Query manually" check box
  • paste the following Query, where pay attention on the string describing which AppPool we want to use as trigger. For the Internal Web Site, we use LyncIntFeature and for External - LyncExtFeature.

<QueryList> <Query Id="0" Path="System"> <Select Path="System"> *[System[Provider[@Name='Microsoft-Windows-WAS'] and (EventID=5074)]] and *[EventData[Data[@Name='AppPoolID'] and (Data='LyncIntFeature')]] </Select> </Query> </QueryList>

  • Click OK on this window.
  • Click "Action" tab.
  • Click "New"
  • Type powershell.exe in "Program/script" box.
  • Add the following in "Add adgument (Optional):" box -ExecutionPolicy Unrestricted c:\Scripts\MeetExtWarmUp.ps1 where MeetXXXWarmUp.ps1 is the the name of the script we prepared earlier for the appropriate AppPool.



Complete the setup. your final configuration will look similar to mine.


What we just configured:

  1. Monitor the System Event Log for EventID 5074 (indicating AppPool recycle event"
  2. Examine the event to determine which AppPool was just now recycled
  3. Fire up the appropriate PowerShell script to sent fake meeting join request, thus forcing the server perform Meeting Join procedure (the server is not aware this is not a real meeting ID)

Step 3: Verify if our scheduled task works as expected.

Upon scheduled AppPool recycle, we find the corresponding entries as shown


Also, when examine the IIS log, we expect to see appropriate entries, indicating the meeting join request was invoked and precessed.


In conclusion, it is possible to "warm-up" the app pool of Lync Server Web Site to avoid meeting join delay.

This article can also be used as reference as of how to capture and evaluate specific event and execute Scheduled Task based on the input.

Monday, November 11, 2013

Lync 2013 client and and pictures from custom URL

I was very excited to find that custom picture form URL feature is back!

With this function available once again, the Administrator can enable it in order to solve an issue that haunted many users for quite some time – users might have set Custom Picture URL which is no longer accessible or the picture is not recent.


Because the functionality was removed at some point, such users got stuck with old picture and could not change it.

To enable “Show picture from a web site” on Lync 2013 client, two conditions must be satisfied:

  1. Lync 2013 client updated at minimum to September 2013 Lync client update (currently we are at November 7, 2013).

  2. Administrator must enable it via Client Policy.
$pe=New-CsClientPolicyEntry -Name EnablePresencePhotoOptions -Value True
$po=Get-CsClientPolicy -Identity Site:NorthAmerica
$po.PolicyEntry.Add($pe)
Set-CsClientPolicy -Instance $po


Note that the example above modifies Site policy “NorthAmerica”. We cannot update all policies in "bulk" - must be done one at the time. To update the Global Client Policy:

$pe=New-CsClientPolicyEntry -Name EnablePresencePhotoOptions -Value True
$po=Get-CsClientPolicy -Identity Global
$po.PolicyEntry.Add($pe)
Set-CsClientPolicy -Instance $po


We find the new Policy Entry created:


After restart, Lync client will receive the new setting via inband provisioning:


Thursday, July 11, 2013

July 2013 Lync Users Group Meeting

I am proud to announce the third quarterly Lync User Group meeting schedule for July.  I will be speaking at the Atlanta event July 31st.

Two topics will be discussed:

SIP Academy

  • This session will be presented by AudioCodes and the focus is on “What is SIP (Session Initiation Protocol)”, how Microsoft Lync utilizes SIP, and how to read and understand Microsoft Lync SIP logs.

Lync as PBX Replacement

  • Co-existence and migration scenarios for Lync as PBX Replacement, as well as real life experience shared by local UG members.

Here is the list of scheduled events for this period. To participate, simply sign-up for any event by creating Meetup account (if you don’t have one already) and confirm your attendance for one or more events.

DateLocationAddress
July 16th
3:30 PM
Cincinnati UC User GroupMicrosoft Office
4605 Duke Drive, Suite 800
Mason, OH 45040
July 16th
5:30 PM
Detroit UC Users GroupMicrosoft Technology Center
1000 Town Center Drive, 19th Floor
Southfield, MI 48075
July 25th
6:00 PM
Chicago UC Users Group Microsoft Technology Center
200 East Randolph Drive, Suite 200
Chicago, IL 60601
July 25th
3:00 PM
San Francisco UC Users Group Microsoft Office 835 Market Street, Suite 700
San Francisco, CA 94102
July 30th
6:00 PM
Nashville UC Users GroupMicrosoft Office 2555 Meridian Blvd, Suite 300
Franklin, TN 37067
July 31rd Atlanta UC Users GroupMicrosoft Technology Center
1125 Sanctuary Parkway, Suite 300
Alpharetta, GA 30009
August 1stWisconsin UC Users GroupMicrosoft Office N19 W24133 Riverwood Dr, Suite 150
Waukesha, WI 53188
TBANew York UC Users GroupMicrosoft Technology Center
1290 Avenue of the Americas, 5th Floor
New York, NY, 10104



See you on 31st.

Wednesday, June 19, 2013

Configure Fiddler for Lync Mobile sign-in troubleshooting



While we have the online utility (https://www.testexchangeconnectivity.com) to verify our Lync Mobility deployment’s functionality, sometimes the Lync administrator must deep dive into the sign-in process. Below I will describe a basic setup of both Fiddler and mobile devices for troubleshooting the process or… just plain fun, if you want to see what is happening under the hood.

Download and install Fiddler2 (http://fiddler2.com/get-fiddler). I always use the latest stable build… Do not start Fiddler after installation. There is one more executable we must install in order to enable tracing for iOS devices. Download and install the New Certificate Maker (http://fiddler2.com/r/?FiddlerCertMaker).

With both installed, start Fiddler and go to Tools -> Options



Go to the Connections tab and check “Allow remote computers to connect”. Click OK on the pop up note.



 The next step is to enable HTTPS Decryption. Go to the HTTPS tab and check “Decrypt HTTPS traffic".



Apparently Eric Lawrence has a good sense of humor – “Scary text ahead”, eh…

When set to decrypt HTTPS traffic, Fiddler works in (sort of) SSL Offload mode. Fiddler will make HTTPS requests from the local machine’s applications (or remote clients if it is configured to act as HTTP Proxy) re-encrypt with the “Fiddler Root Certificate” and then pass it to the application/client who made the request. This is because the Lync client for example, makes true HTTPS request and expects to receive answers via the established SSL channel. Without re-encryption, Fiddler would pass the answer from our reverse proxy as HTTP and the Lync client would say “Ughm, this is not what I expect”.

In a new window will you will be asked you to confirm the Fiddler root certificate installation. Click "Yes" to confirm on TrustCert confirmation screen.


At this point Fiddler is capable of decrypting local HTTPS traffic for further review. However, our goal is examine traffic between Lync and the mobile clients.

Right now, the newly generated Fiddler Root certificate is not present on the mobile device I'm going to test with. The first step is to export the certificate via the Fiddler Options screen.


The certificate can be now found on the desktop.


Because I will direct traffic from my phone to pass on TCP port 8888 (the default port Fiddler uses for HTTP proxy) via the machine Fiddler is installed. I will create a firewall exception from PowerShell. Note: You must run PowerShell as Administrator to create the rule.


Import-Module NetSecurity
New-NetFirewallRule -DisplayName "Fiddler Proxy Exception" -Description "Allow remote client to use Fiddler" -Protocol TCP -LocalPort 8888 -Enabled True -Profile Any -Action Allow -Verbose


Now I will email the certificate to myself, and then install it on my Windows 8 phone and iPad.



Once the email arrive on my WP8 device, I open it, click on the attachment and confirm the installation.




Next - configuration of the phone to use HTTP proxy for web requests. Select your Wi-Fi network...



...slide Proxy to ON, enter the IP Address of the workstation where Fiddler is installed and set the port (8888 if you used default).


To test the setup, I use Bing on the phone and search for “Lync 2013”. The expectation is Fiddler will show traffic coming from the phone.


Yup, it works! Now I will sign-in with my Lync 2013 Windows Mobile Client and watch the magic.


Oh joy. Clicking on a line will reveal the “under the hood” process.

Here are few screenshots of the iPad configuration steps.The steps are similar taken on WP8 device.

Install Fiddler Root Certificate:





Configure the Proxy:





Refer to your manufacturer's instructions how to setup other devices.

This setup can be used to trace and review sign-in process of both Lync Mobile 2010 and Lync Mobile 2013 clients. Fact is - I advice to do so and get familiar with the differences between MCX and UCWA services.

Oh, you can use Fiddler to troubleshoot Lync MX as well.

Install the EnableLoopback Utility (https://www.fiddler2.com/dl/EnableLoopbackUtility.exe)

If you have Lync MX already installed, when Loopback Utility screen opens, locate and check Lync app.

That's all, folks... Enjoy.