Monthly Archives: December 2016

XenDesktop EDT over NetScaler – Benchmarking

With build 11.1, 51 Citrix released support for EDT over NetScaler. Which I have blogged about before which excels in situations where latency is high, packet loss is at a minimum.

EDT Networking deep dive –> http://msandbu.org/benchmarking-adaptive-transport-for-hdx/

Enabling EDT in NetScaler –> http://msandbu.org/enabling-remote-hdx-enlightment-data-transport-in-netscaler-11-1-build-51/

NOTE: That Mac Based Forwarding which also cripples Framehawk also cripples EDT, so you need to disable MBF in order for it to work. There are also some other scenarioes where EDT doesn’t work

NetScaler Gateway Double-Hop No
NetScaler pure LAN proxy No
NetScaler GWaaS (Gateway as a Service) No
NetScaler to VDA DTLS encryption No
HDX Insight No
NetScaler Gateway in IPv6 mode No
NetScaler Gateway SOCKS No
NetScaler Gateway Multi-Stream ICA (MSI) No

Now the scenarios where EDT works

NetScaler Gateway Yes
NetScaler Gateway with High Availability (HA) Yes
NetScaler Gateway with High Availability (HA) optimization Yes
NetScaler with Unified Gateway Yes
NetScaler with GSLB Yes
NetScaler with Cluster Yes
Citrix Receiver to NetScaler Gateway DTLS encryption Yes
Dual Secure Ticket Authority (STA) on NetScaler Gateway Yes
NetScaler Gateway ICA session timeout Yes

 

So therefore I’ve decided to do a benchmark of regular ThinWire over TCP vs UDP to compare the differences.

So just to start with some networking benchmarks running video over regular latency (Which is 10 – 15 MS) latency
UDP (EDT) 12 MS latency

image

TCP  12 MS latency

image

UDP 100 MS latency

image

TCP 100 MS latency

image

So from what we can see there is a a drop in bandwidth usage on the TCP side. Because of the latency I get retransmissions and I get delays and window scaling takes some time before it catches up.
Now here is a video which shows the difference between TCP / UDP on 10 MS and 100 MS latency

The upper left (TCP 10 MS) upper right (UDP 10 MS)
Down left (TCP 100 MS) Down right (UDP 100MS)

One issue I have as of now is that file transfer is not working really well….

EDT with 100 MS latency
ICA-UDP-100MSFILE

EDT with 12 MS latency
ICA-UDP-FILE

DNS Security with NetScaler

With NetScaler build 51, Citrix introduced DNS Security Options which allows for simple defining security policies against DNS endpoints on NetScaler. The DNS Options are split into different options.

  • Cache Poisioning Protection
  • DNS DDoS Protection
  • Manage Exception (whitelist / blacklist) servers
  • Prevent random subdomain attacks
  • Bypass the cache
  • Enforce DNS transactions over TCP
  • Provide root details in the DNS response

dns.PNG

These options can now be found under Security –> DNS Security, so let’s dig a little deeper into what the different options here mean.

  1. Cache Poisioning Protection (Cache poisoning is a type of attack in which corrupt data is inserted into the cache database of the DNS server so for instnace if we have GSLB based ADNS servers running on our NetScaler (This feature is enabled globally already)
  2. DNS DDoS Protection (Specify that hte NetScaler should drop it is reciving alot of requests coming from the same endpoint within a short time range, where the action is that the request could be blocked or warn (which send a SYSLOG event)
  3. Whitelist/Blacklist Clients Is pretty much a reponder policy which specifies who is allowed to query the DNS server
  4. Prevent Random Subdomain Attacks option directs the DNS responder to drop DNS queries that exceed a specified length.
  5. Bypassing the cache option directs the NetScaler appliance to bypass the cache for specified domains, record types, or response codes when an attack is detected.
  6. Some DNS attacks can be prevented if the transactions are forced to use TCP instead of UDP, so with this feature we can specify records or domains which should only be available over TCP instead of UDP
  7. In some attacks, the attacker sends a flood of queries for unrelated domains that are not configured or cached on the NetScaler appliance. lf the dnsRootReferral parameter is ENABLED, it exposes all the root servers.The Provide Root Details in the DNS Response option directs the NetScaler appliance to restrict access to root referrals for a query that is not configured or cached. The appliance sends a blank response.

You can find a youtube video of the process of setting it up and seeing the policies in Action here –> https://www.youtube.com/watch?v=mWlY0hyA6KU

But note that these settings do not use any of the mode advanced features such as DNSSEC, but mostly resolves around leveraging Reponder policies to block attacks.

Remote Access to XenApp Express / Essentials

Now even if the XenApp Express “Essentials” hasn’t been released yet, there are some tiny bits of information available, and the piece I’m going to talk about is remote access. So let us picture the regular Citrix Cloud setup where we have our management plane in Citrix Cloud and we have our NetScaler (or NetSCaler Gateway on-premises)

So all communication between our end-users and our Desktop / Servers are being proxied via the NetScaler. Now if we look at the Citrix Cloud pricing we can notice that ICA Proxy VPX’s (Two Platform licenses per subscription so we can setup a HA pair) are included as part of the pricing https://www.citrix.com/products/citrix-cloud/buy.html

Now managing a NetScaler is a bit time consuming and requires some specialized knowledge to setup and maintain. It also requires a digital certificate and an public IP of sorts to endpoints can communicate with it externally.

Now there is also an other option called NetScaler Gateway Service (aka HDX Proxy)

hdxproxy.PNG

Which replaces the use for NetScaler for remote access purposes, and it also gives a more simplified setup.

hdxproxy2.png

NetScaler Gateway Service is in essence a Windows Server which runs on the Cloud Connector server and it communicates directly with Citrix Cloud to proxy ICA traffic between the endpoint and the backend VDA.

And this does not require any other firewall openings, public IP’s or certificates besides the ones already configured for the cloud connector

One thing to be aware of is this service was initally setup only in San Jose so all ICA traffic from Receiver was tunneled there and back again, but Citrix have now implemented Geo based load balancing and clients will therefore be routed to the closest AWS location where NetScaler Gateway in Citrix Cloud is running.

NOTE: The price listed about is 9$/month PER user

  • The serviceonly provides HDX traffic as part of the XenApp and XenDesktop Service. Other NetScaler Gateway functionality will not work like Smart Access and such
  • The Citrix Cloud Connector located within your Citrix Cloud resource locations communicates with Citrix-run cloud services over the Internet. Currently this communication channel does not support authentication at outbound proxies for access to the Internet.
  • All network traffic is protected by SSL, but to provide the NetScaler Gateway functionality, HDX traffic is present in memory in an unencrypted form on the CloudConnector VM
  • To use the NetScaler Gateway Service, you must use StoreFront hosted within Citrix Cloud.
  • No HDX insight capabilities

However, configuring the setup for NetScaler Gateway service is pretty simple, you can do it by going into Citrix Cloud –> Manage –> NetScaler and define NetScaler Gateway enabled and choose cloud hosted NetScaler Gateway Service

image

So when this is enabled you will notice a service spinning up on your cloudconnector VM which enables the remote access

hdxproxy3.PNG

Now with NetScaler HDX Proxy enabled you just have to configure you VDA servers / agents an no longer need NetScaler VPX instances. The only downside is that this in all cases will not provide the same low latency and does not scale as high as regular NetScaler does.

SSL and TCP Insight on NetScaler MAS

With the latest release of NetScaler MAS 11.1 build 51, Citrix now has some cool new enhancements to the Insight Module which allows us for instance to monitor SSL Transactions/Ciphers/Protocol and such under Web Insight so we can see more in detail how SSL is behaving and such.

We also have some new enhancements with TCP Insight which allows us to see more in detail how TCP is behaving, so for instance it will allow us to check in more detail how a TCP profile is affetcing the performance.

So what do we need to have this feature enabled?

Now most of this is automatic setup when you add an instnace to MAS and define Insight for a couple of vServers that are SSL based which we want to monitor.

  • AppFlow enabled for a Virtual Server and make sure they are licensed correctly from within MAS. Have a forwarder defined to MAS
  • UFLD enabled and a forwarder defind, this setting you can find under System –> Auditing –> ULFD Servers and define the MAS
  • The following enabled on AppFlow settings on the MPX/VPX you want to monitor from
    appflow.png
  • Last piece of the puzzle is to enable TCP insight on NetScaler MAS, which is a setting under System –> Analytic Settings –> Configure Features, and mark the checkbox under “Enable TCP Insight”

So now we just have to make some requests to the website and see what happens of AppFlow collection.

appflow2.PNG

SSL Insight looks like this, it displays SSL information inside Web Insight on a server level. But we also have some more stuff stached away in the report pane which allows us to see more SSL transactions based.

And lastly there is TCP insight, which allows us to see TCP bandwidth going between Internet and Radio.

appflow3.PNG

It can also show upstream/downstream of differnet interfaces and VLAN’s. Since this is my lab enviroment there isn’t much information to show.

appflow5.PNG

Now this is a new feature, but I would like for Citrix to implement a form of “Configuration Advice” for TCP insight based upon what kind of traffic, bitrate and latency end-users are expecting. In stead of just showing stats.

 

What’s new in NetScaler 11.1 build 51 + NetScaler MAS

Today Citrix, released new anticated versions of the NetScaler and NetScaler MAS firmware, which is now on build 11.1 build 51, and contains a lot of cool new features!

What’s new in NetScaler 11.1 Build 51
In this version of NetScaler it has some major enhancements in terms of features.

* Intune support (Integrates with Intune Conditional Access to deliver access to on-premises applications) Requires an updated Android or iOS Citrix SSL VPN app
NOTE: iOS VPN app was updated back in November which came with support for Intune)
* Enhanced Transport Protocol support( Citrix now supports EDP on NetScaler which came in tech preview on XenDesktop 7.12, more info on setting it up here –> http://msandbu.org/enabling-remote-hdx-enlightment-data-transport-in-netscaler-11-1-build-51/)
* DNS Security enhancements (Comes with a prebuilt wizard to setup security rules for DNS servers on NetScaler)
* Role-based access within a partition
* Force Password Change (Meaning that the system user nsroot needs to change its password first time after logging in)
*  DNS resolution through NetScaler Gateway or the NetScaler G ateway Plug-in if IPv6 is enabled on the client’s adapter and the ISP provides the IPv6 DNS address on the IPv6 stack now works!
* Support for Jumbo Frames on NetScaler VPX Appliances Running on AWS
* Support for PCI Passthrough Interfaces on NetScaler VPX Appliances Installed on VMware ESX Server

Now there are also numerous bug fixes in the firmware as well.

What’s new in NetScaler MAS 11.1 Build 51
Citrix has did a major uplift on MAS with alot of new features there as well, especially on the Insight part, which I intend to follow up with some blog post later.

* View start and end time of a terminated user sessions in Gateway Insight (This is now available as a report and under the users pane yay!
* SSL Insight which provides integrated, real-time monitoring of secure web transactions
* TCP Insight (monitors the metrics of the optimization techniques and congestion control strategies (algorithms) used in NetScaler)
* Support for Monitoring HAProxy Instances ( We can now use NetScaler MAS to monitor HAProxy instances in our infrastructure.
* Numerous integration options with OpenStack (Newton, Mitaka)

There are also mulitple enhancements to the dashboard an such, so stay tuned for a blogpost going more in-depth on some of the enhancements.

Enabling Remote HDX Enlightment Data Transport in NetScaler 11.1 Build 51

So with the recent release of NetScaler 11.1 – build 51, Citrix released support for Enlightend Data Transport which is a new feature in XenDesktop 7.12 which now also supports NetScaler in this new release. In order to leverage the protocol there are a couple of things we need to enable in order to use it.

XenApp and XenDesktop 7.12 or higher (required to enable the feature using Studio)
VDA for Desktop OS 7.12 or higher
VDA for Server OS 7.12 or higher
StoreFront 3.8
Citrix Receiver for Windows 4.6 (download Receiver 4.6 here –> https://www.citrix.com/downloads/citrix-receiver/windows/receiver-for-windows-latest.html)

  • Add firewall rules to allow inbound traffic on UDP ports 1494 and 2598 of the VDA.
    • Note: TCP ports 1494 and 2598 are also required, however they are opened during the installation of the VDA. In this release, 1494 and 2598 must be manually enabled for UDP.
  • IPv4 VDAs only. IPv6 and mixed IPv6 and IPv4 configurations are not supported.

In Studio, enable the policy setting, HDX Enlightened Data Transport (it is disabled by default).
To enable the policy setting, set the value to Preferred (or Diagnostic mode), then click OK.

  • Preferred. Enlightened Data Transport over UDP is used when possible, with fallback to TCP. No additional configuration is required to optimize for LAN and WAN conditions.
  • Diagnostic mode. Enlightened Data Transport over UDP is forced on. Fall back to TCP is disabled.
  • Off. TCP is used. Setting to Off does not impact other features which use UDP (for example real-time audio transport or Framehawk.And lastly we need to allow the transport method to Citrix Receiver as well, so there we need to configure the group policy setting
    1. Computer Configuration > Administrative Templates > Citrix Receiver > Network Routing > Transport Protocol for Receiver.

image

    1. Set the policy as needed: Enabled (EDT first with fallback to TCP) or Off (TCP).
    2. Select the Communication Protocol for Citrix Receiver for Windows as Preferred, On or Off.
      Now we need to configure the NetScaler as well. First thing we need to do is enable DTLS support on NetScaler
      image
    3. And lastly you need to rebind the Certificates on the virtual gateway server, and then we can connect.

      So how can we check that its working properly? As by default Citrix uses TCP so if you run a CTXsessions from CMD you will see that TCP is the default shown before we apply the policy, afterwards it should show like this
      UDP –> CGP –> ICA
      image

      There are somethings you should remember in this build however. it does not support double hop, HDX insight, nor IPv6 mode

Setting up WebSocket access on Citrix NetScaler

When setting up my internal Rancher Master Service I noticed that I was getting some wierd timeout values in the UI, the management console acted gray and it went extremly slow when accessing it externally. After starting up Developer mode in Google Chrome I noticed right away that I was getting some websockets timeouts.

WebSockets Connection to “ws://” failed error during websocket handshake: Unexpected response code 404

68747470733a2f2f6265656d6170702e73332d65752d776573742d312e616d617a6f6e6177732e636f6d2f37306233626166312d653132662d346530642d396664612d6638373233386338643333652e6a7067.jpg

When I tried to connect to the Rancher master server internally I noticed that things were working alot smoother then when I was connecting to it externally.

I noticed pretty quickly that it was the NetScaler which was not allowing WebSocket connections as part of of the virtual server that I had configured.

Now there are something that you need to be aware of, WebSockets is not equal to HTTP, it is an entirely different protocol, so in my case I had configured a HTTP based load balanced vServer which of course did not regonize the websocket request that my client made to the Rancher Master server.

Now this is of course something that is easy to fix on a NetScaler. First we need to configure a HTTP Policy, which is found under System –> Profiles –> HTTP Profile, the easiest thing is the mark the default profile and click add.

Inside the profile settings window, there is only one setting we need to define

websockets.PNG

That is the “Enable WebSocket” connections, which allow WebSocket connection over HTTP based vServers. After we have created this HTTP profile we just have to bind it to a virtual server.

Go into Traffic Management –> Load Balance –> Then into the virtual server we want to enable websockets, and click edit. Choose Profile on the right side and click edit on HTTP profile

websockets2.PNG

And voila you are done! WebSockets is now enabled over existing HTTP load balaced virtual servers.(Note: Storefront HTML5 based Citrix client also uses WebSockets)

Microsoft Intune and Citrix NetScaler integration

So with the upcoming release of the NetScaler 11.1  build 51, it will now finally support Intune integration which I have been waiting on for some time now. This new feature allows for Conditional access against on-premises web applications like SharePoint and such.

So for instance the integration allows NetScaler to pull compliance data from Intune, enabling conditional access policies. The conditional access policies give NetScaler Gateway to control the  access based on device functionalities and so on. For example, an administrator can create a policy where only devices with “Camera” disabled are granted access.

NOTE However: Only iOS and Android clients are supported at this time, and it requires an updated VPN client.

So how does this integration work?

localized image

Source: http://bit.ly/2iFjcon (Citrix)

  • An device is enrolled to Microsoft Intune
  • Policies and applications are publised to the endpoint
  • A users tries to connect to an on-premises web application
  • User is redirected to NetScaler Gateway website
  • The User presents an Oauth token to the Authentication Policy on the NetScaler Gateway
  • If the device is successfully enrolled, access to the on-premises web application is granted.
    • If the device is not enrolled, the VPN client will display an error message with a link to the Intune page to enroll the device
 So there are some things to note here however, this integration does use a Oauth authentication policy, so the user presents an Oauth token from Intune to the NetScaler Gateway vServer (which is configured with a non-addressable AAA vServer which handles the Oauth authentication)
In an upcoming post (after the build is released) ill write how to setup the integration and with a video showing how it works.

TCP Fast OPEN – Citrix NetScaler

TCP Fast Open (TFO) is a TCP mechanism that enables speedy data exchange between a client and a server during TCP’s initial handshake. By using the TFO mechanism, you can reduce an application’s network latency by the time required for one full round trip, which significantly reduces the delay experienced in short TCP transfers.

So how does it work? This picture describes it alot better!

Source: http://image.slidesharecdn.com/devconf2014-kernelnetworkingwalkthrough-140304102610-phpapp01/95/devconf-2014-kernel-networking-walkthrough-16-638.jpg?cb=1393929597

It is important however that we need to have a supported client and a supported server to make this feature work. This feature was introduced in NetScaler 11.1 as it just needs some configuration to be able to work properly.

This can be done by adjusting a TCP Profile with the TCP Fast Open value

image

We can also define how long the TCP cookie should be used, by default this is set to zero (Which is defined in the TCP parameters on the NetScaler

image

After this setting is configured we need to enable TCP fast open for Microsoft Edge. Note that this feature is not enabled by default. Microsoft wrote a blog about TCP fast open earlier this year –> https://blogs.windows.com/msedgedev/2016/06/15/building-a-faster-and-more-secure-web-with-tcp-fast-open-tls-false-start-and-tls-1-3/

But not everything is well documented in the blogpost! first of you need to have 1607 build to get suppor for TCP fast open in the Windows Kernel. If you have TCP fast open you can see that enabled by using this command

netsh interface tcp show global (You will see TCP fast open) if you do not see it present you need to update your Windows 10.

image

To enable TCP fast open  in Edge you need to open Microsoft Edge (Using build 14352 or higher) and type

about:flags

Then scroll down and enable TCP fast open, then restart the browser.

image

Next we need to test this that it is working! by default in Microsoft Edge  it ONLY WORKS UNDER HTTPS/TLS it makes sense but it is not documented.

Here we can see from WireShark the client request going to the web-server
(10.217.215.153 = Windows 10 client, 10.217.215.223 = NetScaler Virtual Server)

image

And here I can see the NetScaler responding with the Cookie

image

And here we can see that the client uses the TCP Open Cookie for second request

image

So voila! So will this small chance improve web performance? No yet! There are still a number of ISP which blocks the TCP Fast Open cookie header in TCP (ref: https://www.simula.no/file/conext2015pdf/download) which means that it falls back to regular TCP and then triggers a TCP retransmission.

But for those that have TCP fast open enabled on their web-servers, as seen here implementing TCP fast open will allow for fast download of websites

Source: https://aeckert93.files.wordpress.com/2014/06/screen-shot-2014-06-02-at-10-41-48-pm.png

Citrix NetScaler and Action Analysis to limit user connection to a website

So this is something that I was tasked with a couple of weeks ago, where a customer was having issues with their network bandwidth to their website was exhausted because of a sophisticated DoS attach which seemed like a regular HTTP request, but it made the webserver exhaust its resources and comsume bandwidth upstream so it limited authentic users to access the website, now this was a web-server issue but while the application vendor was fixing the issue I needed to fix the issue.

Now they were using Citrix NetScaler as a load balancer,  but I’ve never been able to use it for this particular purpose, so I was first thinking about HTTP DoS but that didn’t help to mitigate the task. So what I ended up with was feature I’ve only worked with once or twice which is Action Analysis.

So what we wanted to do was to limit a end-user’s bandwidth for one hour so it wouldn’t cross the limit we have set, this way we wouldn’t stop the authentic users while stopping the attackers.

Now in order to use this properly we need to configure a Stream selector which defines the endpoint we want to monitor,

stream_01

We also need a Stream identifier (where we bind the selector) and if we want monitor bandwidth, connections and so on.

stream_02

Lastly we create one responder policy first, where we have one policy which is used to collect statstics for the Stream identitier. Here we need to specify Action NOOP since we just want it to collect stats and not do anything. The Expression is defined to ANALYTICS.STREAM(“CLIENT_BW”)COLLECT_STATS

stream_03

Now lastly we need to bind this policy to virtual server where we want to collect information. We also have the option to see the statistics that are gathered. Note however that since we specified 1 hour in the Stream Identifier this will refresh every 1 hour. We can see statistics by going into AppExpert –> Action Analytics –> Stream identifiers –> Stream sessions.

stream_04

And from here we can see the statistics. Which is collected for that vServer session.

stream_05

Now we can also specify a second responder policy which triggers when a user has exeeded the bandwidth limit we specify, in this case we want to limit it too 100MB only, so we then bind the same policy to the same vServer where we bound the first policy.

stream_06

Easy, peasy 🙂