Why is Office 365 so slow?…..
Yesterday Microsoft released a new tool to help with the performance assessment of Office 365 which you can find here –> https://o365networkonboarding.azurewebsites.net/ Now this tool is initially aimed at locating the closest Service Front Door for Office 365 to help with guideance on how traffic should flow.
Service Front Doors are essentially are points-of-presence where traffic for Office 365 is terminated against Microsoft. When setting up Office 365 for an organization there are many factors that apply for optimizing for the traffic and much can go wrong and affect the network badly and deliver a bad user experience.
Looking at it left to there are many settings that can affect performance.
-
Device
If the device does not have enough compute capacity to handle the different Office Applications or having a non-optimized performance it can affect performance. Same goes with if you have other service or applications on the endpoint such as Antivirus that constantly looks into the network traffic as an IPS it can also affect performance. Also having for instance laptops with different power modes can also affect performance.
-
Browser
Are all browsers the same ? No… There are many new changes that have been implemented server side (and will be implemented moving forward. Such as TLS 1.3 (Which is not implemented on Office 365 yet) which reduces the MS it takes to establish handshake in a TLS session and the same with TCP Fast Open which was only supported on certain browers when it was available service side. Also Browers behave differently when it comes to page rendering and support for new standards to it is important to look at the system requirements (https://support.office.com/en-us/article/which-browsers-work-with-office-online-ad1303e0-a318-47aa-b409-d3a5eb44e452)
-
Local Network
This is an important piece. If you have a powerful machine and running the latest verison of Google Chrome with no services running locally on the machine that should affect performance for web based applications you are good to go right? Well not if you are sitting on a crappy Wifi with constant packet loss or non optimized wifi card on the different computers, just consider how this will affect skype/teams video and audio meetings. Other parts that can affect performance is components such as Forward Web Proxies, VPN which limits MTU/MSS or other disabled configurations.
-
Connectivity
This is of course the most important factor on how traffic flows from end-users across the WAN to Microsoft Service Point. Here we need to ensure that traffic is optimized and using the most optimal path directly to the closest location. In many enterprise you have multiple locations worldwide where you might have a central VPN termination point where all outbound traffic is handled, this will affect performance badly since end-users in other regions will be terminated at a non-optimal service and that traffic needs to traverse the VPN configuration. Of course Office 365 needs to live side-by-side with other traffic on the WAN that you have established and we need to ensure that some traffic is prioritized and guaranteed some bandwidth. If you want to get some insight on what the latency should be like, this link at Verizon can give you some insight –> https://enterprise.verizon.com/terms/latency/
I’ve previosly written about deployment of Office 365 in RDSH and VDI enviroments https://msandbu.org/guide-to-deploying-office-365-in-rdsh-and-vdi-enviroment/ which focuses on configuration of the different components. In this part I want to highlight a bit more the network piece here to optimize the performance for Office 365.
Optimizing Network for Office 365
-
Optimize DNS Resolution
Office 365 consists of 100+ of different DNS entries which the browser and the different Office ProPlus applications needs to query/lookup, and optimizing DNS is one of the first steps on how you can optimize Office 365. Take a look at the different providers here and you can see the difference between then –> https://medium.com/@nykolas.z/dns-resolvers-performance-compared-cloudflare-x-google-x-quad9-x-opendns-149e803734e5
It is also important to know that if you have a DNS server hosted in another region your endpoint might be routed to another Office 365 datacenter, ensure that DNS servers are located within the same region as your clients. Also, ensure that DNS requests are not taking long to process. Try doing nslookup outlook.office365.com and ensure that you get a response that is part of your geographic region, can be verified using https://www.iplocation.net/
-
Bypass Network Proxies and Traffic Inspection devices
Microsoft also recommend buypassing local forward web proxy solutions (Bluecoat, WebSense and such) for outgoing connections to Office 365. Of course the forward web proxy solutions are there to implement security measurements for internet traffic, they serve little purpose for Office 365, since it already has numerous security measures.
-
Local Network Breakout or Split tunneling for VPN users
Another issue is if you have a multiple locations which are connected in a hub-spoke network where are VPN traffic is terminated in a central site which again is the source for outbound traffic to the internet. This adds latency and can potentially redirect traffic to a geographically distant location. You should always plan to have the traffic go directly to O365 instead of having to travel unecessery paths. If you need to have VPN you should configure split tunneling for Office 365.
-
Plan for having adequate outbound bandwidth and IP addresses
Office 365 can consume a lot of bandwidth for outgoing connections, and it can also consume a lot of outbound ports in a DNAT/SNAT Configuration. Microsoft has determines that behind a single public IP you can host about 6000 devices if only using Office 365 based upon the port usage from Office 365 perspective.
• 2000 «Heavy» users using Online mode in Outlook About 20 Mbps at peak
• 2000 «Heavy» users using Cached mode in Outlook About 10 Mbps at peak
• 2000 «Heavy» users using audio calls in Skype/Teams About 110 Mbps at peak
• 2000 «Heavy» users working Office using RDP About 180 Mbps at peak (With UDP traffic trough RDP Protocol)
NOTE: Securing traffic and activity to Office 365 requires something else than traditional security products such as proxies and firewalls. Take a closer look at CASB Solutions which have direct API integrations with Office 365 to provide security (Such as Microsoft Cloud App Security)
-
Other Network Performance Issues –
This is mostly looking into some of the more in-depth features in the TCP Stack which might affect performance against Office 365.
Windows Scaling Factor: Without Window Scaling, TCP only has 64KB for a TCP receive window. This means that I client can only transmit 64KB worth of data before having to stop and wait for acknowledgments (ACK) to be returned by the host. This issue becomes much more apparent on high latency (long) connections in which more data can be placed on to the line before an ACK can be returned.
SACK and TCP MSS
SACK, (DSACK and FACK) which makes it easier to resume a TCP stream in event of packet loss, since it does not need to retransmit all packet which have been lost in a TCP session.
TCP MSS, the maximum segment size, is a parameter of the options field of the TCP header that specifies the largest amount of data, specified in bytes, that a computer or communications device can receive in a single TCP segment.
The two pictures below shows the difference from a computer that doesn’t have SACK and Lower TCP MSS compared to the other which has max MSS and Sack enabled.
So how can I check this from my computer? The simplest way is to check using WireShark or Message Analyzer, Open up a TCP Session and look for the
- TCP Sack (Should be enabled)
- MSS Value (Should be 1460)
- Window Scaling (Should be enabled)
TCP Idle Time
Firewalls are normally designed for internet access to Web Pages. This means TCP Sessions were not expected to be idle for a long time, but it no longer the case for SaaS based services. If there were any idle TCP Sessions, the firewall simply closed them. Users were not affected by this using web pages only, but now the situation is different and this can especially affect clients like Outlook.
Summary
Many have already implemented Office 365 without considering the effect that it has on the network. Historically Exchange, SharePoint and Lync/Skype was hosted inside the datacenter, when we now are moving all the “heavy” services a thousand kilometers away, it will affect the performance and end-user experience so therefore it is important to optimize any “bumps” in the road.
So no Office 365 is not slow, its most likely the network..