Last week I encountered a lot of unknown traffic aimed at our of Citrix ADC / NetScalers at one of our customers, first I was thinking that someone has managed to compromise our instance, but early findings didn’t see anything. Microsoft also reported that there was a lot of unknown traffic going outbound on UDP 443 form the Netscaler Gateway.
Then I checked social media, and seeing that no one else so far had seen anything I quickly posted this
After a while I got contacted by a bunch of people worldwide which were seing large scale attacks with the similar traffic patterns against DTLS enabled NetScaler / ADC Gateway Servers. Now since many have alreay written a lot about the attack, and that Citrix has notified this issue and is working on a fix, I wanted to write a bit more in-depth about the DTLS protocol and some of the current issues that it is facing and what I belive that Citrix is working on. Hopefully also to provide some more guidance about this protocol as well.
DTLS itself has been around for a couple of years, for Citrix enviroments it has been important for EDT (Enlightend Data Transport) which marked the shift from TCP to UDP based transport protocol for Citrix enviroments.
Just to start with a few highlights on what DTLS is and what it is aimed at solving.
1: DTLS is a protocol to provide TLS over datagram transport (UDP) the issue is because that TLS itself does not handle packets that may be lost or reordererd
2: The TLS handshake layer assumes that handshake messages are delivered reliably and breaks if those messages are lost.
3: DTLS is in two versions, v1 and v1.2 (there is no version 1.1) this to provide similiar versioning with TLS. DTLS 1.0 corresponds to TLS 1.1, and DTLS 1.2 corresponds to TLS 1.2.
So DTLS is deliberately designed to be as similar to TLS as possible, both to minimize new security invention and to maximize the amount of code and infrastructure reuse.
Now when Citrix introduced the EDT protocol they only supported to support DTLS 1.0 given the support from the VDA’s (OS) at the time)
- Windows 8 and Windows Server 2012 introduced support for DTLS 1
- Windows 10, version 1607 and Windows Server 2016 added support for DTLS 1.2
With the initial version of EDT. Citrix only supported DTLS on the front-end of the NetScaler, meaning that traffic from the client —> netscaler was using EDT over DTLS but the backend was not. Then Citrix introduced DTLS support using v1 end-to-end and now As from XenApp and XenDesktop 7.16 VDAs, DTLS is supported (depending on the backend VDA) using DTLS 1.2 (From release Netscaler 13.0 build 47.x, DTLS 1.2 protocol is supported in addition to the earlier supported DTLS 1.0 protocol.)
By default vServers are set at DTLS 1.0
set vpn vserver vpnvserver -dtls off
add vpn vserver vpnvserver_dtls dtls 10.108.45.220 443
bind ssl vservser vpnvserver_dtls -certkeyName sslcertkey
set ssl vserver vpnvserver_dtls -dtls12 ENABLED
The benefit is that end-to-end DTLS security is now supported: Receiver => NetScaler Gateway => VDA.
There are however some issues with datagram protocols and that is that they are extremely susceptible to a variety of DoS attacks.
1: An attacker can consume excessive resources on the server by
transmitting a series of handshake initiation requests, causing
the server to allocate state and potentially to perform
cryptographic operations.
2. An attacker can use the server as an amplifier by sending
connection initiation messages with a forged source of the
victim. The server then sends its next message (in DTLS, a
Certificate message, which can be quite large) to a victim
machine
Now both these types of attacks can be mitigated, using a DTLS cookie. So when the When the client sends its ClientHello message to the server, the server
can respond with a HelloVerifyRequest message (it is optional as part of the rfc)
So the cookie works like this:
When responding to a HelloVerifyRequest connection, a client must use the same values as it did in the origional ClientHello (version, random, session_id, cipher_suites) The server uses these values then to generate a cookie. The Cookie that is generated is unique per IP and is generated based upon the following values.
Cookie = HMAC(Secret, Client-IP, Client-Parameters)
If the HelloVerifyRequest is not enabled, it means that each connection the server will present with ServerHello connection and the Certificate packet, which is a lot larger then the clienthello connection.
When the second ClientHello is received, the server can verify that the cookie is valid for that given IP address. Now then it means that the spoofed IP address must reply the HelloVerifyRequest session and since they are in most cases forged, the session will not be initiated.
This is a setting available on the NetScaler / ADC today using the command that can solve it.
set ssl dtlsProfile nsdtls_default_profile -helloVerifyRequest ENABLED
NOTE: This setting does not provide any defense
against DoS attacks mounted from valid IP addresses, but as seen from the other sources it seems like DDoS attacks from spoofed IP addresses.
This is by default set to not enabled. I have seen that this stops the attacks, but the problem is that this feature can make your ADC stop responding because of memory leaks on certain builds. Citrix has now released a firmware update which fixes this –> https://www.citrix.com/downloads/citrix-adc/firmware/release-13-0-build-71-40.html
Also follow the guidelines here to enable the helloverifyrequest as well for the DTLS profiles https://support.citrix.com/article/CTX289674