Private Endpoints – SNAT – UDR and Azure Firewall

When using PaaS services in a hub-and-spoke architecture a best-practice approach is to use Private Endpoints for accessing those services. This allows us to ensure that these services are only available internally in the Azure VNET and not publicly available.

For instance, with this example below. Where we have a private endpoint to a storage account.

Private Endpoints is a virtual NIC (Network Interface Card) which is placed within a virtual network and is attached to a PaaS service.

Now unlike regular network cards, these types of virtual network cards are a bit different. First:

• Private Endpoints only support TCP (not ICMP/UDP)
• They do not support NSG flow logs. So, this means that traffic analysis is not going to work
• Does not support UDR/NSG now (However this is a feature that is currently in preview) Manage network policies for private endpoints – Azure Private Link | Microsoft Docs.

However, there are some other caveats as well that you should be aware of regards to Private Endpoints.

By default, a private endpoint will automatically create a /32 default route which will automatically propagate to the VNET it resides in and other peered VNETs.

This means that if you have a hub-and-spoke model where you have a Private Endpoint in one of the spokes then the private endpoint default route will be also propagated into the hub network as well.

If you check the system routes on one of the endpoints, you will see something like this. It should also be noted that this route is only working for TCP-based traffic.

Azure will route traffic based upon the following longer prefix wins so for instance 10.0.0.0/24 wins over 10.0.0.0/16. If you have a route on the same prefix, Microsoft will use the following priority.

• User-defined routes
• BGP routes
• System routes

Now, what is the problem with this default route? When you have a regular hub and spoke model you want to route all traffic through the central firewall, therefore you have UDRs or route tables that are sent to the Azure Firewall. Now, this causes some issues with Private Endpoints in two different scenarios.

1: Spoke to spoke traffic through the Hub to a Private Endpoint

In this scenario in a typical hub and spoke architecture where services in one spoke need to talk to another spoke to a private endpoint. How does the traffic flow?

It is important to note that the Private Endpoint will have a default /32 route propagated to the VNET where the Azure Firewall resides, so the firewall knows how to reach the endpoint. The Virtual Machine has a UDR attached to the Subnet it resides in which describes how to reach the private endpoint through the Azure Firewall.

Then we have network rules in Azure Firewall that allows traffic from the VM to the Private Endpoint. However, this will not work, why?

Let us look at the traffic flow:

1: Traffic from the VM uses the UDR to point to the Azure Firewall

2: The Azure firewall sees the destination traffic, processes the Network rules, sees a match, and forwards the packets to the Private Endpoint.

3: The private endpoint sees the traffic based on the traffic flow and tries to route the packet back directly to the virtual machine.

4: The VM is not expecting any traffic back from the Private Endpoint and it drops the TCP session.

Why is this? because unlike other services it looks at the flow and not the source MAC address which would link the traffic back to the Azure Firewall. This is also a limitation that is described under private endpoints

So how do we solve this? 

Use Application rules in Azure Firewall instead of Network Rules. This will apply SNAT to the firewall traffic for the private endpoint instead of packet forwarding. If we use Application Rules  the traffic flow would look like this

1: Traffic from the VM uses the UDR to point to the Azure Firewall

2: The Azure firewall sees the destination traffic, processes the Application Rules, sees a match, and initiates a new TCP Session to the private endpoint.

3: The private endpoint sees the traffic based on the traffic flow and routes the traffic back to the Azure Firewall.

4: The Firewall sees the traffic coming back from the endpoint and routes the traffic back to the original VM based on the TCP session table.

However, this requires that you have mapped the private endpoint to a DNS record for it to be able to process the application rules.

2: Hub to Spoke traffic to a private endpoint

As I mentioned earlier for all private endpoints you will automatically get a /32 default route for each private endpoint that will be propagated to each VNET and peered VNETS. So, if you have traffic from Hub to a Spoke where the Private Endpoint resides, how do you ensure that the traffic to the private endpoint will be routed via the firewall (if you have that requirement?)

Well, you have the option to override the default route and create a custom UDR with the same prefix /32 to route the traffic to the central NVA. However, there is a limitation to having so many routes. However, this is what is currently in public preview to provide the ability to override private endpoints with a larger prefix –> Public preview of Private Link UDR Support | Azure updates | Microsoft Azure