I’m seeing high latency and packet loss to/from AT&T across multiple ISPs. I can’t find anyone else talking about it anywhere, so is anyone else seeing it?
ATT <> ATT cross country seems to be fine.
I see it from MidCo in Fargo, but not Midco in KS, Comcast and Cox.
Cloudflare’s Radar shows lot of BGP activity on one of the primary ATT ASNs for today.
Cloudflare Radar: https://radar.cloudflare.com/as7018?dateRange=7d
Screenshot: https://imgur.com/xca7nAe
No ETR, but understand there to be an aerial Charter fiber in Dallas being repaired. Knocked down the AT&T to TELIA peer from 3x100 to 1x100. I’m on Midco KS at home and have about 35% packet loss to our IPs that AT&T advertise for us (also in Kansas).
Also seeing issues between ATT and cox.
ATT to Cloudflare/Google and most public web seem to be okay, but connections to cox clients are failing. >75% ping failures.
We’re seeing it for one office in particular right now, 65% loss on their VPN tunnel for anyone on an AT&T network at home.
It’s been very rough today for a variety of our connections in Central US, especially in the Kansas City and Nebraska areas.
BGP and Routing issues are so painful. I try to check Thousand Eyes Outages Map and see what noise is showing up on DownDetector (yes, I know DD is crowd sourced noise).
And then see what you can get through your trace routes. You need a couple different data center / ISP / customer locations you can trace through if you want anything actionable.
The issue is that once you confirm something, what can you really do about it? I’ve had bad peering last for almost 2 weeks before, even on an escalation from a data center ISP.
We’ve purposefully routed an entire customers’ WAN traffic through one of our data center routers before in the past to get around this fresh hell.
Sympathies for any of you dealing with this right now…
We’re seeing it in Columbus, OH right now, but all our issues are AT&T to AT&T. We have a number of VPN issues reported by remote workers this morning, and so far the common denominator is that they’re all on an AT&T connection and so is the VPN server. Connecting from other ISPs/hotspots where we can seems to confirm this as well.
We are seeing it as well from Tempe AZ ( Cox ) to Jupiter FL ATT
We are in OKC and remote employees using AT&T are dropping packets to our primary COX internet connection but we have an AT&T connection at our DR site that is working for them. This has been ongoing for at least the past 5 hours. (8am to 1:30pm CST)
So we agree that there is some sort of backbone issues between AT&T and COX at least.
Oh man - this was a plague for me last week. I had all sorts of issues with it at one of my larges clients. AT&T didn’t see any issue on their end. But, my the Unifi Dream Machine Pro said otherwise. Power loss at the client’s office resetting EVERYTHING, and putting the ONT in passthrough mode is the ONLY thing that made a difference. Cox to AT&T traffic was an absolute crapshoot this morning as well.
This is resolved for me in DFW this morning.