So today I was asked to help troubleshoot an issue where an customer had setup a new pair of NetScaler MPXs which was setup using LACP and different VLANs, after the initial setup and started with a basic load balancing setup for Storefront 3.1, stuff were not working.
The service was working as it should, meaning that the SNIP traffic was working to the backend server. Opening a HTTP connection to the service didn’t do anything.. Even the networking tools in IE and Chrome didn’t see anything. I tried to do a ping but that was working fine
Then my first idea was WTF? Why isn’t HTTP traffic working but ICMP was working…
So like any good IT-guy we started at the bottom of the chain (Physical and data link layer)
Was the networkin working as it should?
VLAN configured properly? Check
LACP configured properly? Check
Routing properly configured? (Mac based forwarding didn’t work either)
After looking over the configuration, we noticed that outgoing traffic was going to one MAC address and then responding from another MAC address, which might pinpoint the issue, but that was a false positive since it was just HSRP protocol doing it thing…
Now since PING was working we just wanted to verify that all the other parts of the network was working as it should, and of course we didn’t see any firewall issues as well. Now comes the interesting part, we setup WireShark on a RDS server to see if the HTTP traffic was actually going back and forth to the NetScaler.
And it was, we saw the HTTP traffic going back and forth as it should, so looks like the network was working as it should.
Problem was now that the HTTP traffic packets were coming back to the client, but was NOT appearing in the browser and this seemed to happen all clients that tried to connect, and again
So what is happening now?? Then I get the information that they have an HTTP proxy solution in place. Which of course could be the culprit but this was not the case….
Now last night I actually saw a blog from Citrix here –> https://www.citrix.com/blogs/2016/02/24/announcing-storefront-3-5/
AHA bingo! one quick look at the event log on the Storefront server we saw this.
So then we logged into the SSL parameter settings of the Service Group and saw this, after we disabled TLSv12, it worked!
So the solution was actually in the first lines of the blogpost… MPX and Storefront 3.1. Funny thing is that we actually had another similar case with SDX and VPXs where a consultant has upgraded their VPX appliances frmo 10.5.55 to 10.5.59, what happend is that the VPXs started to communicate using TLS 1.2 backend with older servers, which then caused the same problems. After upgrading the VPXs to the latest 11.64 version they were able to DISABLE TLS 1.2 backend for the SSL parameters and things started to work again.