Microsoft has traced last week's global Microsoft 365 outage to an undetected July software defect that rendered over 80% of its affected networking gateway servers unresponsive.

Microsoft Says Over 80% Of Gateway Servers Failed During Global Microsoft 365 Outage

The420 WebDesk
7 Min Read


Follow us on Google News


Follow The420.in on Google News

Microsoft has revealed that a software defect introduced in July remained undetected for more than three months before triggering last week’s global Microsoft 365 outage, leaving more than 80% of servers supporting its networking gateway unresponsive and disrupting email services, businesses and government operations worldwide.

Charles Lamanna, Microsoft’s Executive Vice President of Copilot, Agents and Platform, disclosed on Monday that three interconnected technical factors caused the disruption that began on November 12 and continued into the following day.

The findings, released as part of Microsoft’s preliminary post-incident assessment, provide the clearest explanation yet of how a previously undiscovered software defect overwhelmed the company’s infrastructure and defeated safeguards designed to prevent widespread service failures.

Microsoft has apologised to customers and said a detailed post-incident review will be released later this week.

FCRF’s Chief AI Officer Certification Returns on November 14 to Prepare Future AI Leaders

Software Defect Remained Undetected For More Than Three Months

Microsoft traced the disruption to a software modification introduced in July specifically for the networking gateway supporting several Microsoft 365 services.

The gateway acts as an intermediary between users and Microsoft’s online applications, directing incoming requests towards the infrastructure responsible for processing them.

According to Lamanna, the software modification contained a defect that could cause servers to become overwhelmed under particular operating conditions.

The problem involved contention within an internal software process, where competing operations attempted to access shared computing resources. Under heavy demand, this prevented affected machines from responding normally.

The defect remained unnoticed despite being deployed internationally for more than three months.

Microsoft acknowledged that its testing procedures failed to identify the weakness because they concentrated on less powerful hardware. The defect became more severe on newer servers equipped with a greater number of processing cores.

This meant that machines expected to provide greater computing capacity were particularly vulnerable when the conditions necessary to trigger the defect emerged.

Microsoft clarified that the affected software was designed specifically for its networking gateway infrastructure and that the underlying defect does not apply to broader Windows installations.

Three Factors Triggered A Cascading Global Failure

The software defect alone had not caused any noticeable disruption until changes in network traffic and available computing capacity created the conditions for a widespread failure.

Approximately 15 hours before the outage, Microsoft Teams traffic was redirected from an alternative networking gateway to the primary gateway supporting Exchange Online, the company’s cloud-based email platform.

Although such transfers form part of Microsoft’s routine operations, the change increased demand on infrastructure already operating with reduced capacity.

Nearly 14% of the gateway’s server capacity was unavailable because of an unrelated technical problem from the previous day.

When traffic increased during peak operating hours in the United States, the combination of higher demand and reduced capacity exposed the hidden software defect.

Servers began accumulating requests before becoming unresponsive. Microsoft’s traffic management systems automatically redirected users towards functioning machines, but this increased pressure on the remaining infrastructure.

Instead of containing the disruption, the redistribution of traffic caused additional servers to encounter the same defect.

Within less than two hours, more than 80% of the affected gateway’s servers had become unresponsive.

Microsoft also disclosed that a separate problem prevented its infrastructure isolation mechanisms from containing the disruption, allowing the failure to spread internationally.

Why Outlook And Exchange Online Took Longer To Recover

The incident exposed differences in the recovery capabilities of Microsoft’s individual applications.

Services including Teams, OneDrive, SharePoint and Office applications were able to shift away from the affected gateway. Teams messaging and meetings remained available for much of the disruption.

Exchange Online, however, could not use the same alternative recovery route.

As a result, Outlook users faced prolonged difficulties accessing mailboxes, sending messages and receiving emails across web and desktop applications.

Engineers encountered further complications when attempting to restart affected servers. Machines returning to service were immediately confronted with accumulated requests and frequently became unresponsive again.

Microsoft acknowledged that its automated recovery mechanisms were not designed to address a failure affecting such a large proportion of its infrastructure.

The age of the underlying defect also complicated recovery. Conventional procedures for reversing recent software changes could not quickly remove a modification that had been deployed several months earlier.

Engineers eventually developed an update that restored the affected software to its earlier functioning state. Groups of servers were temporarily removed from operation, updated, restarted and gradually returned to service.

Microsoft subsequently confirmed that the affected services had been restored.

Indian Businesses Face Questions Over Cloud Service Dependence

The outage disrupted businesses across more than 70 countries and territories, including airlines, financial institutions, healthcare organisations and government departments.

In India, IndiGo, Air India and Akasa Air reported difficulties with email-based customer support, while financial services companies experienced problems accessing research and communicating with clients.

Air India temporarily redirected its email communications to maintain contact with passengers. Flight operations were not reported to have been affected.

Globally, financial institutions and other businesses were forced to rely on alternative communication arrangements, including telephone calls and traditional messaging systems.

The disruption has renewed concerns about organisations depending on a single technology provider for multiple essential business functions.

Microsoft’s latest findings also demonstrate that having additional computing capacity does not necessarily guarantee resilience when different components share a previously undetected software defect.

The company has ruled out a cyberattack as the cause of the outage. Its forthcoming detailed assessment is expected to provide further information about the failure of protective mechanisms and the measures required to prevent a recurrence.

What This Means For You:

Businesses relying on cloud-based productivity platforms should maintain independent backup communication channels and regularly test their continuity plans. Organisations should also ensure that essential customer support and operational functions remain accessible during prolonged outages affecting their primary technology providers.