Saturday, November 3, 2012

Lync Mobility on Dual Homed Front-end and co-located Mediation



One statement in this (http://technet.microsoft.com/en-us/library/hh690037.aspx) article always intrigued me.


Small organizations might elect to deploy single standard edition Front End server with co-located Mediation role and in many cases will have two network interfaces – one bound to Lync registrar services and one – dedicated to PSTN (Mediation). While this is normal (and in many cases preferred configuration), with the introduction of support for Mobile devices, we quickly found that mobility simply does not work as expected.

I was able to replicate this behavior in my Lync 2013 lab as well. My front end server have two interfaces – 10.255.2.101 set in Topology as “Primary” and 192.168.1.101 set as “PSTN”. My goal was to isolate traffic to and from the PSTN gateway on its own subnet.


In the above configuration, the failure of mobility support is obvious – mobile client was able to sign-in, but presence is shown as “Away” and while I can change it from the mobile client (where the new status will be shown correctly to my contacts, mobile client itself will not update. Also, IM works from mobile client to other parties, but not vice versa.

The logical conclusion is that we have one way communication. However, the sign-on is two way process and we were able to sign in, with other words we DO have two way communication. Very confusing, indeed.

Mobile client uses the web services to sign-in and communicate with Lync server. Reverse Proxy forwards external traffic to the HLB VIP, which in return forwards to IIS on the FE pool, which by Topology definition is bound to the Primary (10.255.2.101 in this case). 


Not surprisingly, we have two way communication because the traffic always flows between well-known IPs.

After we are signed to the mobile client, any modality, however, is processed by the SIP Stack. Examining the log from failed IM attempt from federated contact shows something strange:


Hmm, connection was closed? Indeed, because the Peer is… 192.168.1.101 i.e. out interface dedicated to PSTN.


This is not Lync server problem! It is how Windows OS is handling the precedence (Metric) of the network interfaces. While we have INVITE to the pool


…we called the wrong IP.

Using utility named dnstest.exe, we see something very interesting. When use the utility with option –w (wire query only), DNS query for the second member of the pool returns the LAN IP only, which make sense, because DNS have only one A record, pointing to the LAN IP address (PSTN IP is not registered).


Same query for the host itself returns the two IP addresses on the server and note – it NEVER makes DNS wire query (confirmed with Wireshark).


Furthermore, the “wrong” IP is listed first in order, which explains why when the server initiates call to the pool:5087, it does so calling the wrong IP address.

Looking the Network Adapter settings, we note that the two are set to “Automatic Metric”.


Here I set the Primary adapter with Metric 1


…and the PSTN adapter with Metric 99


Now dnstest tool shows the desired order:


Restarted all Lync services on this machine (Stop-CsWindowsService, Start-CsWindowsService), did so on the second server in my EE pool and Mobility worked as expected with all modalities. The Sip service now called the correct IP address:


Conclusion
This example shows that it is possible to configure dial homed Frond end service with co-located mediation role for Lync mobile support. However, this is Lab setup for testing only. There is no official support from Microsoft for this solution and implementing in Production environment is not supported my any means. Use it on your own risk.

Thursday, October 4, 2012

Mediation Server service has encountered a major call completion problem with the proxy peer



I ran to interesting case today. Customer recently implemented ACME SBC gateway in large Lync enterprise environment and SCOM immediately began raising alerts of type “Alert: The Mediation Server service has encountered a major call completion problem with the proxy peer"; with cause: “Calls to this Proxy peer failed 5 times with failure final responses.”

Lync SIP Trace of failed call showed “488 Invalid Media” which on first looked was strange, since not all calls were exhibiting this behavior. However, while examining the SDP, we’ve noticed the INVITE of failed call SBC offered had two “a=fmtp” attributes as shown.



A successful call includes one “a=fmtp” attribute only, as shown.



We could not establish a pattern at this moment (failed calls were observed from different Area Codes and caller ID and so, while the service provider should address the inconstancy, the immediate fix would be to create a rule on SBC to catch if “a=fmtp:101 0-16” is included in the provider’s offer and either remove it, or rewrite to “a=fmtp:101 0-15”. I personally would remove the extra “a=fmtp” and “a=ptime:20” lines, but both should work.

Thursday, September 20, 2012

Configuring Apache as a Reverse Proxy for Lync Server

Introduction

As stated in this blog: http://blogs.technet.com/b/server-cloud/archive/2012/09/12/important-changes-to-forefront-product-roadmaps.aspx, a number of Forefront-branded solutions are discontinued from further releases.

For us, as Lync administrators, including TMG 2010 in this group is extremely painful, having in mind the ease of which Lync external web services were published. This is not the end of the road, however. There are third party software which can be sucessfuly utilized for web publishing.



Today I will install and configure Apache 2.2 in my lab as Reverse Proxy for Lync 2013.
I will be using Windows Server 2008 R2 for this exercise.The server where I will install Apache is not member of domain, because it will be exposed on internet and I want to enhance the security. OS was patched with the latest Windows updates prior to Apache installation.

I will use two virtual network interfaces:
“LAN” – 10.255.2.52/24 and “DMZ” – 192.168.1.52/24
Only DMZ interface have Default Gateway. Routing to the internal subnets via LAN interface was added manually. I will use the same procedure used when configured the Edge server.
LAN interface:

DMZ interface:

Current routing table:


***In this case, Reverse Proxy's LAN interface is on the same subnet where Lync servers are. I can afford it, because this is lab. In production, however, I would consider placing the LAN interface on "internal DMZ subnet" in order to have a phisical separation.

To add static route to internal subnet, ROUTE command was used:
 ROUTE ADD -p 10.255.1.0 MASK 255.255.255.0 10.255.2.1

Now my routing table looks like this:



The DMZ IP address was mapped to Public IP address and the External Firewall was configured with ACL to allow connections from any source to this IP address on port TCP/443. Visit to http://www.whatsmyip.info confirmed my configuration

The first step is to download Apache Win32 Binary including OpenSSL from this link: http://httpd.apache.org/download.cgi where I will download httpd-2.2.22-win32-x86-openssl-0.9.8t.msi. ***See Aditional Resourses at the end of this article.
The installation of Apache is pretty straightforward. Run the .msi and follow the screens…
















I did not change the default installation path and so, the bits were installed to “C:\Program Files (x86)\Apache Software Foundation”.
Before I test my installation, I want to make sure the service will bind to the correct (DMZ) IP address. To do so, I will modify “httpd.conf” file located in “C:\Program Files (x86)\Apache Software Foundation\Apache2.2\conf” directory. Actually, most of the configuration is done in this file and so, I created shortcut on my desktop for easy access.
Once the file open in Notepad, locate the line “#Listen 12.34.56.78:80” and add below:
Listen 192.168.1.52:80


***Note that at this point I am configuring Apache to serve requests for HTTP traffic only. This is because I want to test the general functionality. Also, I have not deployed certificate yet, thus HTTPS protocol will not work at this point. Because Reverse Proxy will be used to serve requsts from Public Internet, I bound the service to the DMZ interface only.
Once the file is modified and saved, restart Apache service.

 
From within the Apache server, open Internet Explorer and navigate to http://192.168.1.52 (the IP address we configured Apache to listen to).

 


Before test from Public internet, we must add Firewall exception for the desired ports and protocols in the Windows Firewall (local machine). I will use PowerShell to do so.



netsh advfirewall firewall add rule name="Apache Port 80" dir=in action=allow protocol=TCP localport=80

netsh advfirewall firewall add rule name="Apache Port 443" dir=in action=allow protocol=TCP localport=443


 


Quick test from a computer on Public Internet shows that my configuration is now operational. At this point, we've accomplished a couple of very important tasks:








  1. Our Apache server is listening on the internal interface and is responding to simple HTTP queries
  2. Our Apache server is also listening and responding to queries for the Internet.
Now we can proceed with some configuration tasks.

Because we will be using Apache as Reverse Proxy and will serve SSL requests, the corresponding modules must be allowed to load at startup. To do so, some lines in “httpd.conf” file must be uncommented.

***Unix folks are familiar with text file configuration. For those who come from Windows world - Apache for Windows is basically a "port" of Unix binaries to Windows. The text file configuration structiure is preserved thus we do configuration by "uncommenting" (allowing), or "commenting" (disallowing) modules to be loaded and run. For example:


Here "mod_auth_basic.so" will be loaded, and "mod_auth_digest.so" will not be loaded.
  1. Locate and uncomment “LoadModule proxy_module modules/mod_proxy.so” line.
  2. Locate and uncomment “LoadModule proxy_http_module modules/mod_proxy_http.so” line.
  3. Locate and uncomment “LoadModule ssl_module modules/mod_ssl.so” line.
  4. Locate and uncomment “Include conf/extra/httpd-ssl.conf” line.
  5. Save the file.
***If you bounce Apache at this point, the service will not start. This is because we configured Apache in SSL mode, but no certificate was assigned yet.
I have already requested and received certificate from DigiCert, which I will use with my test setup. The certificate have CN=webext.lynclog.com and includes meet.lynclog.com, dialin.lynclog.com and lyncdiscover.lynclog.com in the list of Certificate Alternative Name list.

***The above certificate was requested during deployment of my Lync 2010 lab. There are, however, new certificate requirements for Lync 2013 publishing listed here http://technet.microsoft.com/en-us/library/jj205381(v=ocs.15).aspx and, as usual, it is highly recomended to follow Microsoft's best practices to assure sucessful deploiment.

The first step is to export the certificate with the private key to .pfx file.

Locate the certificate, right click and go to All Tasks, Export








We have to use OpenSSL to convert our .pfx file two files in format Apache can interpret.

***In the steps below I will use some custom folders for file location. Those folders are not mandatory and you can use any location.
Create folder “Certificates” on C:\ and move the exported (.pfx) certificate there.
Run Command Prompt as Administrator, and navigate to “C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin” (OpenSSL executable is located there).

 
First, use this command to extract the Private Ket from the certificate:
openssl pkcs12 -in c:\Certificates\rp_cert.pfx -nocerts -out c:\Certificates\encr_lynclog.key.pem
 



IMPORTANT: Always type the commands and not copy and paste. Web browsers tend to replace characters and you might run to problems.

Use this command to extract the certificate from the .pfx file:

openssl pkcs12 -in c:\Certificates\rp_cert.pfx -clcerts -nokeys -out lynclog.cert.pem


Lastly, use this command to convert your key file in RCA format:

openssl rsa -in c:\Certificates\encr_lynclog.key.pem -out c:\Certificates\lynclog.key


Navigate to “C:\Program Files (x86)\Apache Software Foundation\Apache2.2\conf\extra” and create new folder named “ssl”. Copy the files "lynclog.cert.pem" and "lynclog.key" from "Certificates" to "ssl" folder.



Next step is to configure the SSL properties of Apache.

Navigate to “C:\Program Files (x86)\Apache Software Foundation\Apache2.2\conf\extra”, locate “httpd-ssl.conf” file and open it with Notepad.

Locate the line "" and add the following line below:

SSLProxyEngine On


Uncomment the line “SSLSessionCache         "dbm:C:/Program Files (x86)/Apache Software Foundation/Apache2.2/logs/ssl_scache"

Comment out the line “#SSLSessionCache        "shmcb:C:/Program Files (x86)/Apache Software Foundation/Apache2.2/logs/ssl_scache(512000)"


Locate the tag “” and make sure the line “SSLEngine on” is not commented.

Now we will specify the location of our certificate and key files. Locate the line “SSLCertificateFile” and modify the path accordingly.


***As stated above, I decided to use " C:\Program Files (x86)\Apache Software Foundation\Apache2.2\conf\extra\ssl" as store for my certificates. If you use different location, make sure the correct path is used.
Locate the line “SSLCertificateKeyFile” and modify the path accordingly.


Save httpd-ssl.conf (the file we were working on) and restart Apache service.
From public Internet, visit the url (https://meet.lynclog.com in this case) and make sure our server work via SSL.



The last step is configure our Apache server to act as Reverse Proxy.
Open file httpd.conf with Notepad. Locate the line “# 'Main' server configuration” and add bellow the following:
ProxyRequests Off
Order Deny,Allow
Allow from all
ProxyReceiveBufferSize 4096
ProxyPass / https://uspool.lynclog.com:4443/
ProxyPassReverse / https://uspool.lynclog.com:4443/
ProxyPreserveHost On
KeepAlive On


...where uspool.lynclog.com resolves to the VIP of my HLB handling the web services of my EE pool.

***Above we instructed Apache to proxy the requests arriving on port 443 from internet to port 4443 on our Lync server. This is equal to "Bridging" term in TMG:



Restart the service again. If everything was configured correctly, we should now see our Dialin and meet pages:









The above configuration was tested with Lync Mobile support (iPhone) and worked as well.


Summary:
This article describes how to install and configure Apache 2.2 to act as Reverse Proxy for Lync 2010 and Lync 2013 web services publishing.
 
Additional resources:


Apache direct download link: http://mirror.metrocast.net/apache//httpd/binaries/win32/httpd-2.2.22-win32-x86-openssl-0.9.8t.msi

***As of this moment, the donload link to the latest (2.2.23) bunaries is broken. You can, however, check here: http://mirror.metrocast.net/apache//httpd/binaries/win32/ for availability. Always use "Latest stable" version. Beta builds should not be used in Production.

Apache WIKI: http://wiki.apache.org/httpd/

Reverse Proxy Publishing (Lync 2010): http://technet.microsoft.com/en-us/library/gg398872.aspx

Certificate Summary for Reverse Proxy (lync 2013): http://technet.microsoft.com/en-us/library/jj205381(v=ocs.15).aspx