Reverse Proxy and Lync/S4B server

Recently, I came back to the TecheNet forums to check on the issues people are experiencing with their deployment. To my surprise, I found a lot of questions about the Reverse Proxy - how it works, why we need one and, of course, a lot of question regarding issues with Mobility. While the Reverse Proxy (RP) is rather simple topic, lit appears people have a hard time making the connection between how Lync/S4B is designed to work and the role of the Reverse Proxy in the deployment.

In this series of posts I will try to decode the mystery of the Reverse Proxy and provide detailed instruction on how to configure KEMP’s load balancer (free edition) to act as the Reverse Proxy.

Let’s start with the concept of the Reverse Proxy. The Lync Director and Frond End servers have two web sites - Internal and External.

But, why TWO websites?

Security

The first, and obvious, reason - security. We notice that at least two virtual web sites are "missing" from the external web site - cscp (Control Panel) and OcsPowerShell. This is because accessing the management layer of our environment from the Internet is really bad idea and Microsoft removed the accessibility from the Internet altogether.

This reason alone justifies the fact we have two web sites, but... this is not all.

Location, location, location

Yes, location can cost you, even in the Lync server real estate. Here is why:

Since we have two web sites that bind to the same IP address, each must be configured to use unique ports. The default ports in the topology are 80 and 443 for the Internal, and 8080 and 4443 for the External site.

Client workstation would query DNS depending of its location (LAN or Internet) and will resolve the fully qualified domain names (FQDN) to different IP addresses. “Internal” implies that it will serve requests from the inside, or internal networks (LAN) and “External” implies requests from the Internet.

Now, let's look a meeting invite:

The link in the meeting invite is https://meet.domain.com/user/meeting_id and this invite will be distributed to all participants. When we send out the invite, there is no way of knowing where the invitees will come from - some might come from LAN, some from the Internet. It is very important to "let the server know" the "entry point" of any web request in order to have an appropriate response. We will see this later.

Requests from the LAN would come to the Internal web site (DNS resolved to internal IP), which is listening on port 443 (or the port defined in the topology). Requests from the Internet will land on the External web s... wait a second - the external web site is listening on port 4443, but we don't have a "special" invite for those who are to join from the Internet with a "special" link like https://meet.domain.com:4443/user/meeting_id - after all we just want the invitees to click and join, period. The unified invite implies that all meeting join requests will be made to port 443.

And, here the controversy begins (all good stories have controversy…). The very first thought of any novice (no offense) is - "Ha!! I can do that with port forwarding on my firewall and this will save me some trouble fiddling (sorry, Southern expression, been living in the Bible Belt for too long) with a Reverse Proxy and such". I am not saying it will not work, but it will cost you.

The two web sites are meant to serve SSL traffic and so, a certificate must be assigned to each.

By design, the certificates configured on our Lync Front End servers will be issued from the internal (Private) certificate authority and will be free. If we just use port forwarding, requests from the Internet will be terminated with the Private certificate which will not be trusted from browsers and/or devices. As a result, users will get "This web site cannot be trusted" prompt or some functionality will not work at all.

If we want to provide the desired seamless end user experience, in such cases (port forwarding), we have one option - purchase (unless you are one of the companies that issues public certificates and you likely are not!) Public certificate for your Lync Front End server. Here is the Subject Alternative Name list of FQDN's included in the certificate for my lab:

14 names (this server alone, would have been 16 if I wanted to use one certificate on all three servers in the pool) and this is a lab! Yes, two paired Enterprise Edition pools with three members each and two SIP Domains, but... just saying. Such a certificate will cost you a fortune.

Oh hail Reverse Proxy!

Here comes the Reverse Proxy. There are many "solutions" out there, and not many are certified or qualified:

In essence, the reverse proxy is software that accepts web traffic on port 443, terminates the SSL session with certificate of our choice (from Public Certificate Authority, thus trusted across the platforms), and “proxies” the traffic back to Lync server on port 4443. In other words, the traffic lands on the External Web Site on the Director or Front End Web Services. This public certificate will have a Subject Alternate Name list that is a lot shorter (and cheaper) or, if you already have a Wildcard certificate for your domain, you are all set - you can use it and don't have to purchase one explicitly for your Lync deployment. (For reference – the reverse proxy is one of the only places in all of your Lync Infrastructure where you can safely – and in a supported scenario – use a wildcard certificate.)

Here is visual demonstration of a response from the Lync server based on where the request comes from. We did not literally tell the server but rather the server determined the location and formulated the response based on which web site (Internal or External) was visited.

Response from lyncdiscoverinternal (request come from LAN).

Response from lyncdiscover (request comes from Internet). Note that all referrals are to the External Web Site FQDN as defined in Topology.

Money, efficiency and common sense

Let's talk about port 8080 on the External web site. I often see people neglecting this service - "Why do I need it, clients would always do the "real" work over SSL anyway".

True, if you support a single SIP domain in your environment. What about two, as in my lab, or 20, or thousands (think Office365). After all, each sip domain would require its own "lyncdiscover" record, perhaps its own "meet" and so on. Imagine how such a certificate would look like (besides the native limitation of max number of names in the SAN list). It is not plausible to assume that every time a new tenant is created on the Microsoft cloud, someone goes out and acquires and amended certificate.

skypeuc.com is an online tenant on Microsoft cloud. As such, lyncdiscover.skypeuc.com is configured according to the best practices i.e. pointing to the cloud. If we make an HTTPS GET request to https://lyncdiscover.skypeuc.com, we get a rather unpleasant surprise:

However, if the request is http://lyncdiscover.skypeuc.com, it works

What is going on here? It is a simple answer - http redirect. When the browser visited Microsoft's Reverse Proxy on port 80, the request was forwarded to the External web site (port 8080) of the infrastructure which responded with:

i.e. "Go to this FQDN and use SSL protocol". Browser did exactly that, verified the identity of the web site...

...and proceed further. And, by the way, as I said above, Wildcard certificate will do just fine...

Hopefully by now you all are convinced that Reverse Proxy is an essential piece in the Lync deployment puzzle and serious efforts must be made to understand the concept, evaluate the options,  and configure the Reverse Proxy correctly in order to provide your users with stable, working Lync/S4B environment in a 100% supported configuration.

In the next article, I will explain in details how Mobility works. Issues with Mobility are perhaps #2 on TechNet forums and requires yet another explanation.

Lastly, I will show how to configure KEMP Load Balancer to act as Reverse Proxy in your Lync/S4B deployment.

Lync Edge: SIP/2.0 488 Compression algorithm refused

***The real issue was that the servers I was working on were... Server 2012 R2 RTM. I never anticipated that the operation system was not updated with OS and Security Updates. I am trying to track down which KB exactly resolved the issue, but no luck so far. If/when i find out, will update this post again.***

I ran to an interesting issue today. Customer replaced Public Edge server certificate (fortunately in lab) and all of a sudden federation broke.

We saw in the edge traces "SIP/2.0 488 Compression algorithm refused". Interesting, isn't it?

Here I have to mention that the customer uses VeriSign as certificate issuer. In the past, the certificates were signed by VeriSign Intermediate Certificate Authority and never had issues with it. The current certificate, however, was signed by "Symantec Class 3 Secure Server CA - G4".

The issuer itself is not a problem. However, this article - https://technet.microsoft.com/en-us/library/gg398066.aspx states: "All certificates must be signed using a signing algorithm supported by the operating system. Lync Server 2013 supports the SHA-1 and SHA-2 suite of digest sizes (224, 256, 384 and 512-bit)... ".

When the certificate was reviewed, we found that it was signed with signature algorithm "sha256RSA".

Federated Lync Edge servers did not liked this Signature Algorithm, resulting "SIP/2.0 488 Compression algorithm refused"

After the certificate was re-issued from Intermediate Authority, where the acceptable (sha1) algoritm was used, federation was re-established.

The moral of the story is - always verify the Signature Algorithm of your public certificates before assigning to Lync services. alway keep OS up to date!

Increase the size of Lync Server Log

A large customer of mine asked how the size of Lync Server Log in EventViewer can be increased. The request made sense for them since so many events are registered on single server per hour that tracking past events older than one day is practically impossible.

The first and (easiest) method that comes to mind is GPO. However, in Group Policy we find only the native Application, Security and System logs properties.

We have three options to achieve this task:

1. Do it manually on each server

***The default value is 16MB. In this example, the log size was increased to 80GB (81920KB)

2. Use wevtutil.exe (http://technet.microsoft.com/en-us/library/cc732848.aspx).

wevtutil sl "Lync Server" /ms:83886080

***Note that here we express the desired log size in Bytes (80MB = 83886080 bytes)

3. The last (and preferable) option is… GPO. But wait, GPO does not have this option?!?
We can use New Registry Item with Update option to push the new value to all Lync servers in our environment.

The “Key Path” is SYSTEM\ControlSet001\Services\EventLog\Lync Server and the Value data is our new log size in Bytes.

This is it. Apply the new GPO accordingly and either force gpupdate or leave the servers to do so natively.

One last note: Log file sizes must be a multiple of 64 KB. If you enter a value that is not a multiple of 64 KB, Event Viewer will round he log file size up to a multiple of 64 KB.

Q3 2014 Atlanta Lync Users Group Meeting anounced

The Atlnata Lync Users Group will be conducting their next meeting Thursday, July 31, 2014. This meeting will include two sessions, the first session will cover Lync Gateways and Session Border Controllers (SBCs).  The second session will dive into Deployment and Best Practices for Quality of Service (QoS) and Quality of Experience (QoE).

Industry Experts will be on site to deliver these presentations and help answer any questions related to Lync Server.

Food, beverages and additional door prizes courtesy of the Lync Users Group and our sponsors.

Please arrive early. There is Visitor Registration process in place.Informational sheet can be downloaded from this link : Coca-Cola Technology Plaza_Visitor Information

You cannot remove Lync 2013 pool from Topology - Users or contacts are still homed on a pool that would be deleted

Recently I’ve migrated my Lync lab to Windows Server 2012 R2 where I’ve build new site and pools and proceeded with decommissioning the old pool.

After moving all users and objects as recommended, I proceeded with removing the pool and publishing Topology.

Bummer! The log showed there are users or objects “still homed on this pool”

I did check checking everything suggested by the log (Get-CsUser, Get-CsExUmContact, Get-CsCommonAreaPhone, Get-CsAnalogDevice, Get-CsRgsWorkflow, Get-CsDialInConferencingAccessNumber, Get-CsAudioTestServiceApplication, Get-CsTrustedApplicationEndpoint, Get-CsPersistentChatEndpoint) and got the correct output – no objects.

After looking in the RTCLOCAL database of a front end server, I did indeed found ONE object

Found the object in AD...

...and turned out to be... LRS.

Ouch - I forgot that about a month ago I did provision Lync Meeting Room (LRS) to test LRS Portal installation.

Ran Move-CsMeetingRoom and… Topo publishing completed.

In conclusion, I am sure Microsoft will update the code to suggest Get- CsMeetingRoom in the Topo log, but meanwhile, if you have LRS in your environment already, make sure you move it prior to decommissioning Lync 2013 pool if it was homed there.

User might experince delay when join Lync Online Meeting.

Recently I’ve worked with large enterprise customer on troubleshooting strange issue with Lync Meeting join. Users from the pilot (50 people) group would report prolonged (up to 30 seconds) delay between the splash screen and the actual Lync Meeting join. A pattern was not observed (although there is one, just not obvious at first) – it would happen to random users, at random time, joining different meeting ID’s on different pools.

In this article I will explain the background of the “issue” and describe possible solution.

When user click Lync meeting link in format https://meet.contoso.com/user/M33T1NG1D , the request is processed by Director Pool member (if Simple URL points to Director), where after Meeting Discovery, Director Web Service will respond with “301 Moved permanently” to the original request. Client then sends new GET request with destination Web Service of the pool where the meeting will be hosted and meeting join completes.

As Lync server is installed on Windows Server, HTTP requests are served by Internet Information Services (IIS). The web sites (Internal or External) and Virtual Directories within the sites are driven by IIS Worker Process and are logically defined within IIS Application Pools.

As we see, not only whole Web sites (Internal or External), but also Virtual Directory within a web site might be configured to run under dedicated Application Pool.

One aspect of IIS Application Pool is “Recycling”. This process is defined in TechNet as “Internet Information Services (IIS) application pools can be periodically recycled to avoid unstable states that can lead to application crashes, hangs, or memory leaks.” When we install Lync server components, a default value is configured - 1740 minutes (29 hours).

When the Application Pool recycles, all items in the memory are released, a new w3wp process is created, configuration read and applied, .NET reloaded and we site is back serving requests. Under normal circumstances, had we had a request for simple web page as https://dialin.contoso.com , user would not experience visible delay because there is no complexity in assembling and serving this page.

However, Meeting Join request requires extensive server side processing, not discussed in this article. In reality, it takes about 22 seconds for the very first meeting join request to be served after Application Pool Recycle. This fact might cause inconvenience under certain circumstances – in my case a very large front end pool (nine pool members) and only 50 pilot users on this pool. Because the pool web services are Hardware Load Balanced, there is great chance that unique user would hit pool member where the Application Pool was just recycled and expertise 22+ seconds delay. Reports of type “It took forever to join the meeting this morning” were pouring in and frustration was growing fast.

The solution to eliminate the delay is rather simple – monitor the System Event Log for EventID 5074, examine the event description for which Application Pool was just recycled, and fire up Scheduled Task to execute PS script which will simulate real meeting join request. The request will perform “Application Pool warm-up” and because we do so immediately after recycle even, chances a real life user to attempt to join after Pool Recycling, but prior to Scheduled Task execution is practically non-existent.

Steps to configure Application Pool meeting join warm-up:

  

Step 1: Create folder “Scripts” on C:\

• Inside this folder, create text file MeetExtWarUp.txt
• The file should contain single line Invoke-WebRequest https://FE_Server_FQDN:4443/Meet?key=A1B2C3D4 where FE_Server_FQDN is the FQDN of your SE or EE Lync front end server.
• Rename the file to MeetExtWarUp.ps1
• Follow the steps above to create second .ps1 file where the text is 
  Invoke-WebRequest https://FE_Server_FQDN/Meet?key=A1B2C3D4

***Note that we use “:4443” for the External Web Site. It will be omitted in the file used for the Internal Web Site.

Step 2: Create Scheduled Task to execute the .ps1 script when AppPool recycle event is detected.

• Start Task Manager as Administrator
• Expand Microsoft/Windows node, right click and select "Create Task"

• Name the task (WarmInternalAppPool in this example)
• Check "Run with highest privileges"
• Click "Change User or Group" button and select SYSTEM, then click OK.

• Select "Triggers" tab.
• Click "New"
• In the "New Trigger" window, select "On an event" from "Begin the task:"
• Toggle "Custom" under Settings
• Click "New Event Filter" button

• In New Event Filter window, click XML tab.
• Check "Edit Query manually" check box
• paste the following Query, where pay attention on the string describing which AppPool we want to use as trigger. For the Internal Web Site, we use LyncIntFeature and for External - LyncExtFeature.

<QueryList> <Query Id="0" Path="System"> <Select Path="System"> *[System[Provider[@Name='Microsoft-Windows-WAS'] and (EventID=5074)]] and *[EventData[Data[@Name='AppPoolID'] and (Data='LyncIntFeature')]] </Select> </Query> </QueryList>

• Click OK on this window.
• Click "Action" tab.
• Click "New"
• Type powershell.exe in "Program/script" box.
• Add the following in "Add adgument (Optional):" box -ExecutionPolicy Unrestricted c:\Scripts\MeetExtWarmUp.ps1 where MeetXXXWarmUp.ps1 is the the name of the script we prepared earlier for the appropriate AppPool.

Complete the setup. your final configuration will look similar to mine.

What we just configured:

1. Monitor the System Event Log for EventID 5074 (indicating AppPool recycle event"
2. Examine the event to determine which AppPool was just now recycled
3. Fire up the appropriate PowerShell script to sent fake meeting join request, thus forcing the server perform Meeting Join procedure (the server is not aware this is not a real meeting ID)

Step 3: Verify if our scheduled task works as expected.

Upon scheduled AppPool recycle, we find the corresponding entries as shown

Also, when examine the IIS log, we expect to see appropriate entries, indicating the meeting join request was invoked and precessed.

In conclusion, it is possible to "warm-up" the app pool of Lync Server Web Site to avoid meeting join delay.

This article can also be used as reference as of how to capture and evaluate specific event and execute Scheduled Task based on the input.

Lync 2013 client and and pictures from custom URL

I was very excited to find that custom picture form URL feature is back!

With this function available once again, the Administrator can enable it in order to solve an issue that haunted many users for quite some time – users might have set Custom Picture URL which is no longer accessible or the picture is not recent.

Because the functionality was removed at some point, such users got stuck with old picture and could not change it.

To enable “Show picture from a web site” on Lync 2013 client, two conditions must be satisfied:

1. Lync 2013 client updated at minimum to September 2013 Lync client update (currently we are at November 7, 2013).
2. 
   Administrator must enable it via Client Policy.

$pe=New-CsClientPolicyEntry -Name EnablePresencePhotoOptions -Value True
$po=Get-CsClientPolicy -Identity Site:NorthAmerica
$po.PolicyEntry.Add($pe)
Set-CsClientPolicy -Instance $po

Note that the example above modifies Site policy “NorthAmerica”. We cannot update all policies in "bulk" - must be done one at the time. To update the Global Client Policy:

$pe=New-CsClientPolicyEntry -Name EnablePresencePhotoOptions -Value True
$po=Get-CsClientPolicy -Identity Global
$po.PolicyEntry.Add($pe)
Set-CsClientPolicy -Instance $po

We find the new Policy Entry created:

After restart, Lync client will receive the new setting via inband provisioning: