Thursday, April 4, 2013

BigIP F5 as Reverse Proxy for Lync Server, Windows 8 and Lync Metro App

I worked on interesting case today. A large enterprise customer began testing the new Windows 8 OS (desktop and Surface) and received numerous reports for sign-in issues with Lync Metro App. The client will spin for a while and go back to login screen. This seemed strange, since Lync 2010 and 2013 desktop client would work just fine, as well as Lync 2010 Mobile. In this post I will not only explain the problem and provide solution, but also share my method of troubleshooting and some techniques as well.

My approach to every problem is half academic, half technical – in this precise order. I don’t know how you guys work, but my brain retains information relevant to specific case only for relatively short period of time, or to be precise, until I move to the next case and so, for this task, I went back to my notes and refreshed the knowledge as of how Lync Autodiscover works. I decided to concentrate on Autodiscover process because Lync Metro App does not honor SRV records. Instead, it relies on Autodiscover to receive sign-in FQDN and/or URL.
  1. Client will try and resolve either lyncdiscoverinternal.contoso.com or lyncdiscover.contoso.com
  2. Client will request https://lyncdiscoverinternal.contoso.com
  3. Autodiscover will send back the Autodiscover service URL (typically Director Web Services)
  4. Client will authenticate and web ticket.
  5. Client will make new request while submitting the web ticket so that the Director can retrieve the user’s home pool information.
  6. Director will redirect the client to the home pool Web Services.
  7. Client will authenticate again and submit the web ticket
  8. Home pool web services (via Autodiscover) will respond with internal and external Lync services for the user’s home pool.
  9. Client (Lync Metro App in this case) will sign to the Registrar or the Edge server just like desktop client would after discover the registrar via SR record.
All right, how that we know how the process works, next step is to identify at which step Lync Metro App fails. I could only use Wireshark for this troubleshooting session. It is possible to use WS to decrypt SSL traffic indeed if… you have the server private key, which I did not in this case. So, the logical step was to just capture sign-in attempt and see if there is something I can hang on.

The first logical step was to examine the certificate returned from the server. I know, I know – all other service are working at the moment but… just in case. There are many ways to examine server certificate:


By using Internet Explorer. Simply visit the web site by using https protocol and review the certificate.




***While this is certificate from my lab, the process used was the same. Earlier today I verified that the required names (lyncdiscover.contoso.com, director_web.contosof.com and home_pool.contoso.com) were present. So, it is not certificate issue!

Alternatively, you can use tool called RUCT (Remote UC Troubleshooting Tool) http://www.insideocs.com/Tools/RUCT/RUCT.htm. Simply start it, go to Certificate Information tab, input the target URL and port, retrieve the certificate and examine it.


Because the issue was Metro App which runs on Windows 8 OS only, my next test will be from W8 machine. I opened IE10 and went to https://lyncdiscoverinternal.customer.com



I received the expected output and when opend it with notepad, everything looked just fine.


So far I’ve established that DNS resolves to correct IP, the certificate presented from the server have the correct names in the SAN list and the web service is functional.

This is the place to mention the fact that this customer uses F5 Hardware Load Balancer not only as HLB but Reverse Proxy for the External Lync web services. I checked with the network team earlier today and found that the current fimware on F5 is version 10.2

With this in mind, my next stop was… you guessed – F5 compatibility matrix (http://support.f5.com/kb/en-us/products/big-ip_apm/manuals/product/apm-clientcompmatrix-11-1-0.html)


Ouch! As one of my favorite standup comedians Russel Peters often say: "Somebody gonna get a hurt real bad".

I was really not sure what to make of this statement “Windows Phone versions 7 and 8, and Windows RT are not supported”. Any way, as stubborn as I am, next logical step was to "wireshak it" and see what’s up.


I see the normal “Client Hello” followed by “Server Hello”, “Certificate , “Server Hello Done” and “Change Cipher, Encrypted Handshake Message”. Absolutely normal handshake, nothing out of ordinary… handshake done via TLSv1. No RST (reset) i.e both client and server are “talking”. Since I don’t have the private key, I cannot decrypt the traffic and so, for now I will assume the client visited lyncdiscover URL.


I see second handshake in the sequence described above. This must be the authentication attempt. Remember, we must authenticate first in order to receive web ticket.

Oops. This is something new. Note that the TLS version the client requested is v1.2


The server responded with Server Hello TLS v1.2 ...


…but then… nothing. The client dropped the connection.

I already knew from my research that F5 at firmware 10.2 does not support Windows 8 (not that it is clear what exactly is not supported…). I also know that TLS v1.0 works fine (when I visit https://lyncdiscover.contoso.com Internet Explorer 10 does not bombs and so, my conclusion was – this has something to do with the interpretation of TLS v1.2 on both sides.

At this point, just for fun, I decided to disable TLS v1.2 on Windows OS level, which I hoped would force Lync Metro App to use TLS v1.0 and examine the SSL traffic again.

Protocols can be enabled or disabled via registry. To make your life easier, here is link to .reg file which will disable TLS v1.2 on your Windows 8 powered device.

After modifying the registry and reboot, Lync Metro App signed immediately. Wireshark trace confirmed the app used TLS v1.0 to communicate with the Web Services via F5 Reverse Proxy and signed successfully. Download, unzip, move the file to your Windows 8, Merge and reboot.

Simply said, we introduce two new registry keys:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Server


To revert to the default security settings when such "fix" will be no longer necessary, delete "TLS 1.2" key and reboot.

By no means I encourage you to reduce the level of security on your operation system. This “fix” is proposed with the sole purpose that you as IT professional can continue testing all features of Windows 8 and/or Lync Server until such “fix” will be no longer necessary.



32 comments:

Anonymous said...

Very nice case!

Anonymous said...

The problem with this one is that Windows update will then be disabled.. for some reason updates cannot be retrieved anymore..

EXARAY said...

Finally KB2973337 (http://support.microsoft.com/kb/2973337/en-us) solves this Problem.
So update and everything will be fine now :)

Chris Hemsworth said...

The article is so informative. This is more helpful. Thanks for sharing

Learn best software testing online certification course class in chennai with placement
Best selenium testing online course training in chennai
Best online software testing training course institute in chennai with placement
magento developer training

Quickbooks Pro Support Phone Number said...

this is really great to know about.

Webroot Geek Squad said...

it is really great to know about. Thanks for sharing this with us.

Quickbooks Enterprise Support Phone Number said...

it was really good to know about. thanks for sharing this.

Shahil said...

Wonderful post and I hope you more updates from your blog. Thanks to you...
JMeter Training in Chennai
JMeter Training
Spark Training in Chennai
Pega Training in Chennai
Power BI Training in Chennai
Job Openings in Chennai
Linux Training in Chennai
Oracle Training in Chennai
Tableau Training in Chennai
Oracle DBA Training in Chennai
JMeter Training in Velachery

Norton Setup with Key said...

it was really good to know about it. Thanks for sharing this with us.

Norton Product key said...

it was really great to know about.

wordpress tutorial said...

Thanks for the Information
wordpress tutorial

smith machinist said...

Norton.com/setup helps you to deploy Norton setup on your computing devices. Here are the steps for deployment of Norton setup without any interruptions.

norton.com/setup

norton.com/setup said...

Norton is a reputed and cost-effective antivirus suite
company which offers protection so that no virus can damage
your computer. It also provides many other products and
services apart from antivirus.

Technical Support said...

Excellent blog. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking.

Visit: webroot.com/safe

Technical Support said...

It's really helpful. Thank you so much. I can study many things from your blog.

Visit: Webroot geek squad download

Joey said...

Great information you shared through this blog. Keep it up and best of luck with your future blogs and posts.
www.norton.com/setup

Joey said...

I am really grateful to the holder of this web page who has shared this fantastic article at this place.
www.trendmicro.com/bestbuypc

Joey said...

Thanks for sharing the information. Your blog has always been a source of great tips.
www.malwarebytes.com/install

Joey said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such a blog.
aol desktop gold download

Joey said...

I am really grateful to the holder of this web page who has shared this fantastic article at this place.
comcast email sign in

www.norton.com/setup said...

www.norton.com/setup – The digital world is mediated through the internet, and it is the main source of different information shared on the internet. In addition to these things, the Internet is also the source of various viruses and online scams

comcast email sign in said...

Xfinity Connect app is popular for its features and functions now available with all new users interface and advance security features.

www.webroot.com/safe said...

This post is very useful to us thanks for sharing this info with us…

www.norton.com/setup said...

That’s wonderful. many things to learn. thanks for sharing

www.trendmicro.com/bestbuypc said...


Thanks for sharing a useful post here.

www.trendmicro.com/downloadme said...

Thanks for sharing such an amazing article, really informative

www.malwarebytes.com/install said...

https://www.installmalwarebytes.org/install/

aol desktop gold download said...


Very valuable information, it is not at all blogs that we find this, congratulations I was looking for something like that and found it here.

Garmin Nuvi said...

You can view all the available maps that are present in your Garmin Nuvi series on your computer. Garmin Nuvi is a great application where you can view available routes for your upcoming destination. Not this, you can mark the important waypoints like gas stations, restaurants, etc. Once you are done with planning, transfer all data to your Garmin GPS device. latest available updates, downloading and installing Garmin Nuvi Update and Garmin Map Update.

Health Lobby said...

Check the requirements for the criminal case law demanded by the Sindh High Court and feel free to consult advocate. Here you can find how to put the case in Sindh high Court.

vé máy bay từ Nhật Bản về Việt Nam said...

Aivivu chuyên vé máy bay, tham khảo

ve may bay di my gia re

vé máy bay hà nội đi hồ chí minh

đặt vé máy bay phú quốc đi hà nội

vé máy bay ra nha trang

vé máy bay đi đà lạt vietnam airline

taxi đi sân bay

Aastha Agarwal said...

Nice article!
We are glad to read your blog, and we totally agree with you, the list you shared here is excellent. Keep sharing such articles. Me and my team at Rankofy PPC Company In Chandigarh and Rankofy SEO Services In Chandigarh will surely use your tips.