Friday, June 26, 2015

Reverse Proxy and Lync/S4B server



Recently, I came back to the TecheNet forums to check on the issues people are experiencing with their deployment. To my surprise, I found a lot of questions about the Reverse Proxy - how it works, why we need one and, of course, a lot of question regarding issues with Mobility. While the Reverse Proxy (RP) is rather simple topic, lit appears people have a hard time making the connection between how Lync/S4B is designed to work and the role of the Reverse Proxy in the deployment.


In this series of posts I will try to decode the mystery of the Reverse Proxy and provide detailed instruction on how to configure KEMP’s load balancer (free edition) to act as the Reverse Proxy.


Let’s start with the concept of the Reverse Proxy. The Lync Director and Frond End servers have two web sites - Internal and External.



But, why TWO websites?

Security


The first, and obvious, reason - security. We notice that at least two virtual web sites are "missing" from the external web site - cscp (Control Panel) and OcsPowerShell. This is because accessing the management layer of our environment from the Internet is really bad idea and Microsoft removed the accessibility from the Internet altogether.




This reason alone justifies the fact we have two web sites, but... this is not all.

Location, location, location


Yes, location can cost you, even in the Lync server real estate. Here is why:


Since we have two web sites that bind to the same IP address, each must be configured to use unique ports. The default ports in the topology are 80 and 443 for the Internal, and 8080 and 4443 for the External site.





Client workstation would query DNS depending of its location (LAN or Internet) and will resolve the fully qualified domain names (FQDN) to different IP addresses. “Internal” implies that it will serve requests from the inside, or internal networks (LAN) and “External” implies requests from the Internet.

Now, let's look a meeting invite:



The link in the meeting invite is https://meet.domain.com/user/meeting_id and this invite will be distributed to all participants. When we send out the invite, there is no way of knowing where the invitees will come from - some might come from LAN, some from the Internet. It is very important to "let the server know" the "entry point" of any web request in order to have an appropriate response. We will see this later.


Requests from the LAN would come to the Internal web site (DNS resolved to internal IP), which is listening on port 443 (or the port defined in the topology). Requests from the Internet will land on the External web s... wait a second - the external web site is listening on port 4443, but we don't have a "special" invite for those who are to join from the Internet with a "special" link like https://meet.domain.com:4443/user/meeting_id - after all we just want the invitees to click and join, period. The unified invite implies that all meeting join requests will be made to port 443.


And, here the controversy begins (all good stories have controversy…). The very first thought of any novice (no offense) is - "Ha!! I can do that with port forwarding on my firewall and this will save me some trouble fiddling (sorry, Southern expression, been living in the Bible Belt for too long) with a Reverse Proxy and such". I am not saying it will not work, but it will cost you.

The two web sites are meant to serve SSL traffic and so, a certificate must be assigned to each.



By design, the certificates configured on our Lync Front End servers will be issued from the internal (Private) certificate authority and will be free. If we just use port forwarding, requests from the Internet will be terminated with the Private certificate which will not be trusted from browsers and/or devices. As a result, users will get "This web site cannot be trusted" prompt or some functionality will not work at all.


If we want to provide the desired seamless end user experience, in such cases (port forwarding), we have one option - purchase (unless you are one of the companies that issues public certificates and you likely are not!) Public certificate for your Lync Front End server. Here is the Subject Alternative Name list of FQDN's included in the certificate for my lab:




14 names (this server alone, would have been 16 if I wanted to use one certificate on all three servers in the pool) and this is a lab! Yes, two paired Enterprise Edition pools with three members each and two SIP Domains, but... just saying. Such a certificate will cost you a fortune.

Oh hail Reverse Proxy!


Here comes the Reverse Proxy. There are many "solutions" out there, and not many are certified or qualified:




In essence, the reverse proxy is software that accepts web traffic on port 443, terminates the SSL session with certificate of our choice (from Public Certificate Authority, thus trusted across the platforms), and “proxies” the traffic back to Lync server on port 4443. In other words, the traffic lands on the External Web Site on the Director or Front End Web Services. This public certificate will have a Subject Alternate Name list that is a lot shorter (and cheaper) or, if you already have a Wildcard certificate for your domain, you are all set - you can use it and don't have to purchase one explicitly for your Lync deployment. (For reference – the reverse proxy is one of the only places in all of your Lync Infrastructure where you can safely – and in a supported scenario – use a wildcard certificate.)


Here is visual demonstration of a response from the Lync server based on where the request comes from. We did not literally tell the server but rather the server determined the location and formulated the response based on which web site (Internal or External) was visited.

Response from lyncdiscoverinternal (request come from LAN).


Response from lyncdiscover (request comes from Internet). Note that all referrals are to the External Web Site FQDN as defined in Topology.


Money, efficiency and common sense


Let's talk about port 8080 on the External web site. I often see people neglecting this service - "Why do I need it, clients would always do the "real" work over SSL anyway".


True, if you support a single SIP domain in your environment. What about two, as in my lab, or 20, or thousands (think Office365). After all, each sip domain would require its own "lyncdiscover" record, perhaps its own "meet" and so on. Imagine how such a certificate would look like (besides the native limitation of max number of names in the SAN list). It is not plausible to assume that every time a new tenant is created on the Microsoft cloud, someone goes out and acquires and amended certificate.


skypeuc.com is an online tenant on Microsoft cloud. As such, lyncdiscover.skypeuc.com is configured according to the best practices i.e. pointing to the cloud. If we make an HTTPS GET request to https://lyncdiscover.skypeuc.com, we get a rather unpleasant surprise:




However, if the request is http://lyncdiscover.skypeuc.com, it works



What is going on here? It is a simple answer - http redirect. When the browser visited Microsoft's Reverse Proxy on port 80, the request was forwarded to the External web site (port 8080) of the infrastructure which responded with:



i.e. "Go to this FQDN and use SSL protocol". Browser did exactly that, verified the identity of the web site...


...and proceed further. And, by the way, as I said above, Wildcard certificate will do just fine...


Hopefully by now you all are convinced that Reverse Proxy is an essential piece in the Lync deployment puzzle and serious efforts must be made to understand the concept, evaluate the options,  and configure the Reverse Proxy correctly in order to provide your users with stable, working Lync/S4B environment in a 100% supported configuration.

In the next article, I will explain in details how Mobility works. Issues with Mobility are perhaps #2 on TechNet forums and requires yet another explanation.

Lastly, I will show how to configure KEMP Load Balancer to act as Reverse Proxy in your Lync/S4B deployment.

16 comments:

Japheth Nolt said...

When creating a certificate request using the Lync Deployment Wizard, you can uncheck the Lync External site when requesting a certificate from the internal certificate authority. Then only check the external site when generating the request for the public CA. When assigning the certs, do the same.

Any thoughts on why this would not be a good idea?

Drago said...

I said - it is one expensive idea :-) If you support multiple sip domains, the price will skyrocket. On another hand, if you have Wild Card cert already, you can use it and don't have to purchase certificate for the Lync web services at all. However, you cannot assign Wild Card on Lync server, hence we need Reverse Proxy.

Japheth Nolt said...

I have used a wildcard certificate on a FE server (just the external site) without a RP and everything seems to work. Am I missing something?

Drago said...

Interestingly, we find here: https://technet.microsoft.com/en-us/library/hh202161(v=ocs.15).aspx the following statement:

Director. Wildcard SAN entry is supported for Simple URLs (meet and dialin) and for SAN entries for LyncDiscover and LyncDiscoverInternal in Director web components.

Front End Server (Standard Edition) and Front End pool (Enterprise Edition). Wildcard SAN entry is supported for Simple URLs (meet and dialin) and for SAN entries for LyncDiscover and LyncDiscoverInternal in Front End web components.

I should have said: Wild Card "combined" certificate is not supported. Good find! However, I still encourage all administrators to consider Reverse Proxy in their deployments.

Japheth Nolt said...
This comment has been removed by the author.
Japheth Nolt said...

I fully agree that a Reverse Proxy should be considered in Skype for Business deployments.
One major benefit is the ability to have multiple services/servers behind 1 public IP address (Lync, Exchange OWA, Office Web App).
The other benefit is having the lyncdiscover A record on internal DNS point devices (smartphones,tablets) to the RP instead of needing to do hairpinning in the firewall.

Drago said...

...One major benefit is the ability to have multiple services/servers behind 1 public IP address...

Precisely, although this is applicable to very small environments.

Agree on the hairpinning as well. Not sure if it is supported, but indeed it works - i will talk about it in the next article coming in few days.

Anonymous said...

Say you are supporting 10 different sip domains like sip.company1.com, sip.company2.com etc, is there any workaround of not having them in the SAN of external certificate.

If we don't have the sip domains in SAN, we can sign-in using SFB clients with an warning but we won't be able to sign-in onto any of the physical devices (Lync phones).

How O365 is able to accomplish this?

Drago said...

Are you referring to certificate on Reverse Proxy or Edge Server?

Anonymous said...

You are correct, I meant Edge server. How is O365 avoiding having individual SIP domains in the SAN of Edge server.

Drago said...

Microsoft hardcoded in the firmware trust to O365 FQDN's.

Anonymous said...

Ahh, that's not fair. :(

Adibus Sholeh said...

Hi Drago,

very informative articel. Just to make things clear. Are wildcard certificates supported for Skype for Business? I will use it for mobility, federation, etc.

I read on Microsoft Technet that there is no support for wildcard entry as the subject name/CN for any role.

Thank you

Drago said...

Correct. If your certificate ONLY have *.domain.com in the CN (Common Name) and nothing in SAN, such certificate is not supported. However, most of the issuers will include *.domain.com and domain.com in the SAN list for free. At least DigiCert does...

Go to https://webaext.skypeuc.com and examine the certificate. It is a wildcard, it is used on Kemp Reverse proxy and I use it to support two S4B EE pools (all modalities) with single IP address. Of course, this is lab and single IP address should not be used in production, but the example proves the concept.

Also, examine the certificate of Lync Online... https://webdir0b.online.lync.com

LALJEEV said...

Hi Drago

Very interesting article. We completed the installation of SfB with 2 X Standard Edition FE servers in Pool, 1 X Director server in Pool (will add more servers one we purchase hardware load balancer), 1 X OWA server and preparing 1 X Edge server. We have F5 as reverse proxy. Here our concern is about limiting public IPs to be used Reverse Proxy & NATing of Edge external interface. We believe 1 IP is needed for Edge NAT (sip.externaldomain.com, av.externaldomain.com, webconf.externaldomain.com), 2 IPs with reverse proxy - first for FE server URLs(lyncdiscover.externaldomain.com, extweb1.externaldomain.com , meetsb.externaldomain.com, dialin.externaldomain.com, sfbowa.externaldomain.com) which will be pointed to first FE server, if it fails then, manually we will point to second FE server, second IP for Director server / pool (skypeonline.externaldomain.com). Also is it possible to configure / publish the edge through reverse proxy?

Thanks in advance

Drago said...

First and foremost - since Lync 2013 Director role is optional. Do you have a special reason to deploy Director?

As for IP addresses (or lack of thereof), look this tread for an idea: https://devcentral.f5.com/questions/host-header-based-routing Basically, you can use one single public IP address to handle ALL web services on multiple pools for multiple Sip Domains. Then, all you have to do is to change DNS A record for Lyncdiscover, meet, and dialin. The rest is native published and does not have to be changed, ever.

Lastly, no - edge cannot be passed via reverse proxy. You can configure edge with single IP address or multiple (three) IP addresses - it is up to you