Fecebook down

2MoreYears · 10-04-2021, 01:57 PM

I think this is the longest it's been down, about going to 2+ hours now ..? some are happy is down, some are going crazy.

Thank goodness for Twitter.

Sorrybrah · 10-04-2021, 03:12 PM

Yea I've been trying to send some PM on messenger/insta and thought my internet is messing up again.

2MoreYears · 09:50 PM

Here's a good article explaining why it is down:

https://www.zdnet.com/article/what-took-facebook-down/

It seems their DNS is down because their BGP service is down, and BGP needs their DNS to retrieve routing tables. It's like the chicken and egg what-was-first question.

The only way to fix this is by physically accessing their servers, but access is controlled via their network which is down, plus the only ones who know how to fix it are probably working remotely from home.

Facebook will be down at least a day imo, they will have to break some doors to gain access to their servers lol

It's their biggest downtime yet and will be one for the books.

Quote:

The old network troubleshooting saying is, when anything goes wrong, "It's DNS." This time Domain Name Server (DNS) appears to be the symptom of the root cause of the Facebook global failure. The true cause is that there are no working Border Gateway Protocol (BGP) routes into Facebook's sites.

BGP is the standardized exterior gateway protocol used to exchange routing and reachability information between the internet top-level autonomous systems (AS). Most people, indeed most network administrators, never need to deal with BGP.

Many people spotted that Facebook was no longer listed on DNS. Indeed, there were joke posts offering to sell you the Facebook.com domain.

Cloudflare VP Dane Knecht was the first to report the underlying BGP problem. This meant, as Kevin Beaumont, Microsoft's Head of Security Operations Centre, tweeted, "By not having BGP announcements for your DNS name servers, DNS falls apart = nobody can find you on the internet. Same with WhatsApp btw. Facebook have basically deplatformed themselves from their own platform."

Whoops.

As annoying as this is to you, it may be even more annoying to Facebook employees. There are reports that Facebook employees can't enter their buildings because their "smart" badges and doors were also disabled by this network failure. If true, Facebook's people literally can't enter the building to fix things.

In the meantime, Reddit user u/ramenporn, who claimed to be a Facebook employee working on bringing the social network back from the dead, reported, before he deleted his account and his messages, that "DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC)."

He continued, "There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures."

Ramenporn also stated that it wasn't an attack, but a mistaken configuration change made via a web interface. What really stinks -- and why Facebook is still down hours later -- is that since both BGP and DNS are down, the "connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical access to the peering routers and do all the configuration locally." Of course, the technicians on site don't know how to do that and senior network administrators aren't on site. This is, in short, one big mess.

As a former network admin who worked on the internet at this level, I anticipate Facebook will be down for hours more. I suspect it will end up being Facebook's longest and most severe failure to date before it's fixed.

jwxie518 · 10-04-2021, 08:15 PM

Our team at FB did a great job pulling this through. Can't disclose much what happened entire throughout, but all i can say is this literally needed ppl physically in the data centers (multiple of them) and we had to pull in everyone in all time-zones. All hands on deck.

We had some shit employees leaking stuff to public which made the investigation harder to coordinate, wasting time on setting up new and private communication channel.

The other good part is internally the company does a lot of drill on almost daily basis in different region and this type of complete outage had been planned - we had runbooks on who to call, where to meet etc. The sad thing is we still have a lot of tooling behind our network and because DNS couldn't resolve, it was nearly impossible for anyone to get onto any server, let alone opening up our internal wiki pages...

BGP and DNS are not "down" per se. That's a gross simplification. We simply stopped announcing our DNS to the the whole BGP network and the reason will be investigated.

Lucky me I didn't have to be 100% available troubleshooting since I am not oncall for one of the critical systems this week, but man, I can feel the heat even by listening to the live zoom call we had until someone fked up with leaks.

Red neck · 10-05-2021, 02:01 PM

Quote:

Originally Posted by jwxie518

Our team at FB did a great job pulling this through. Can't disclose much what happened entire throughout, but all i can say is this literally needed ppl physically in the data centers (multiple of them) and we had to pull in everyone in all time-zones. All hands on deck.

We had some shit employees leaking stuff to public which made the investigation harder to coordinate, wasting time on setting up new and private communication channel.

The other good part is internally the company does a lot of drill on almost daily basis in different region and this type of complete outage had been planned - we had runbooks on who to call, where to meet etc. The sad thing is we still have a lot of tooling behind our network and because DNS couldn't resolve, it was nearly impossible for anyone to get onto any server, let alone opening up our internal wiki pages...

BGP and DNS are not "down" per se. That's a gross simplification. We simply stopped announcing our DNS to the the whole BGP network and the reason will be investigated.

Lucky me I didn't have to be 100% available troubleshooting since I am not oncall for one of the critical systems this week, but man, I can feel the heat even by listening to the live zoom call we had until someone fked up with leaks.

Is you job hiring/ sponsoring EB3 visa ?lol

jwxie518 · 10-06-2021, 07:30 PM

Quote:

Originally Posted by Red neck

Is you job hiring/ sponsoring EB3 visa ?lol

fb screw up on H1B and to get H1B you'd have to first leave USA

If you are asking moving to EU, maybe, but you'd earn less... there

Red neck · 10-07-2021, 04:18 PM

Quote:

Originally Posted by jwxie518

fb screw up on H1B and to get H1B you'd have to first leave USA

If you are asking moving to EU, maybe, but you'd earn less... there

I have 245i so I dont need to leave the country. It would be EB3 sponsorship

dreamer12345 · 10-10-2021, 04:57 PM

Quote:

Originally Posted by jwxie518

Our team at FB did a great job pulling this through. Can't disclose much what happened entire throughout, but all i can say is this literally needed ppl physically in the data centers (multiple of them) and we had to pull in everyone in all time-zones. All hands on deck.

We had some shit employees leaking stuff to public which made the investigation harder to coordinate, wasting time on setting up new and private communication channel.

The other good part is internally the company does a lot of drill on almost daily basis in different region and this type of complete outage had been planned - we had runbooks on who to call, where to meet etc. The sad thing is we still have a lot of tooling behind our network and because DNS couldn't resolve, it was nearly impossible for anyone to get onto any server, let alone opening up our internal wiki pages...

BGP and DNS are not "down" per se. That's a gross simplification. We simply stopped announcing our DNS to the the whole BGP network and the reason will be investigated.

Lucky me I didn't have to be 100% available troubleshooting since I am not oncall for one of the critical systems this week, but man, I can feel the heat even by listening to the live zoom call we had until someone fked up with leaks.

We all know it was cover for the whistleblower leak now nobody's talking about

S	M	T	W	T	F	S
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30

They need you!

Forum links

Resources

Join our list

National calendar of events

June

Fecebook down