Here is what led to the massive Facebook outage

Social networking site Facebook, along with its family of apps including Messenger, Instagram and WhatsApp, Mapillary, and Oculus, became globally unavailable for a period of six to seven hours on October 4, 2021, rendering inaccessible the services that the world’s largest social media network offers to billions of its users. What went wrong? Security experts […]

Social networking site Facebook, along with its family of apps including Messenger, Instagram and WhatsApp, Mapillary, and Oculus, became globally unavailable for a period of six to seven hours on October 4, 2021, rendering inaccessible the services that the world’s largest social media network offers to billions of its users.

What went wrong?

Security experts identified the problem as a Border Gateway Protocol (BGP) withdrawal of the IP address prefixes in which Facebook’s Domain Name servers were hosted, making it impossible for users to resolve Facebook and related domain names, and reach services.

On October 5, Facebook’s engineering team posted a blog post explaining the cause of the outage. During maintenance, a command was run to assess the global backbone capacity, and that command accidentally disconnected all of Facebook’s data centers.

What is BGP? (BOX)

The internet is a network of massive networks known as autonomous systems. The BGP is used to route information from within these massive networks to the rest of the internet. It is essentially a way for these autonomous systems to exchange routing information between each other.

Out for hours!

Instagram flashed a ‘5xx Server Error’ message, while the Facebook site merely told that ‘Something went wrong’.

Inside Facebook, the outage broke nearly all of the internal systems employees use to communicate and work. Several employees said they resorted to talking through their work-provided Outlook email accounts, though employees can’t receive emails from external addresses.

Impact

During the day of the outage, shares in the company dropped by nearly 5% and Facebook CEO Mark Zuckerberg’s wealth fell by more than $6 billion. According to reports, Facebook lost at least $60 million in advertising revenue.

The outage had a major impact on people in the developing world, who depend on Facebook’s “Free Basics” program, affecting communication, business, and humanitarian work.

Million users for Telegram

During the outage, many users flocked to other social networks including Twitter, Snapchat, Discord, Signal, and Telegram, resulting in disruptions on these apps’ servers.

Telegram CEO Pavel Durov reviled that the messaging app added 70 million users when Facebook and WhatsApp went down. Snapchat saw a 20% surge in users during the outage.

Facebook ‘speaks’

Facebook gradually returned after a team got access to server computers at the Santa Clara, California, data center and reset them. Facebook and related services were generally available again.

After their services were back online, Santosh Janardhan, the company’s infrastructure vice-president: “Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt…We want to make clear that there was no malicious activity behind this outage — its root cause was a faulty configuration change on our end.”

Facebook CEO Mark Zuckerberg apologised to the millions of users who faced hours’ worth of disruption in accessing Facebook, Whatsapp and Instagram through a Facebook post.