# massive Rogers outage -- 13 hours so far



## gardner (Feb 13, 2014)

Anyone else on here that is NOT reliant on a Rogers network? I have Rogers based phones that crapped out at 04:30 this morning but that's about it. Curious of folks have tales of woe and agony. I wonder if this will put a stake through the heart of their intentions towards Shaw. SJR.B didn't move significantly different from the mean today though.


----------



## cainvest (May 1, 2013)

Never been with Rogers, they've had big troubles in the past as well as today.


----------



## Money172375 (Jun 29, 2018)

Are these outages (Bell Fibe Tv had one earlier this year for about 12 hours) usually a result of a cyber attack?


----------



## MK7GTI (Mar 4, 2019)

We will find out soon enough it was some sort of hack/ cyber attack. All the way up Yellowknife things are working fine!


----------



## ian (Jun 18, 2016)

My guess would be that it was a firmware issue. Or perhaps some firmware upgrade gone south.

Cannot see them admitting to a cyber attack even if they had one. That would not do anything for their reputation.

One obvious question though. This is an extended outage with no ETA on a resolution. About 14 hours so far.

Why does Rogers not appear to have a Disaster Recovery program?

Or is their disaster recovery program limited to a few talking heads from their their communications, media, and PR organization or executives?


----------



## jargey3000 (Jan 25, 2011)

back!
what'd I miss?!


----------



## Tostig (Nov 18, 2020)

I'm sort of curious how the employees of Rogers actually discovered or concluded that there was a major outage.

Any Rogers employee who were able to be in contact with each other were obviously NOT on a Rogers phone, internet or cell phone. The CEO or other executives must have learned about it on the radio, like CBC and then discussed it when they were all in the same office together because conferences would not have been possible either.


----------



## m3s (Apr 3, 2010)

ian said:


> Why does Rogers not appear to have a Disaster Recovery program?
> 
> Or is their disaster recovery program limited to a few talking heads from their their communications, media, and PR organization or executives?


Nor adequate redundancy

For comms you normally want a backup to the backup of any single failure points

Things like 911 and point of sale networks need to run on something more.. decentralized


----------



## TomB16 (Jun 8, 2014)

Tostig, you might be surprised by how agnostic the engineers who build national fiber networks are. There are quite a few places where one company will be weak on fiber and another strong. It's often far easier and even cheaper to lease wave demands on another optical network than it is to stand up a new service. In many cases, there are sharing agreements. Also, the nature of the Internet is such that ISPs have to connect to each other, as the "inter" portion of "Internet".

I have no idea what the problem is but to be down for so long, even if it's a hack, is surely an embarrassment and an indictment on the operators responsible. I'm not offended by the idea of something being down for multiple days but where is their redundancy? Either this is a failure of design or they have been operating with an impaired network.

Very few network operators have the discipline to troubleshoot and repair a failed device in a national network with the urgency it requires. I've seen problems languish for months. The mantra is, "This isn't a problem. There is another path." Meanwhile, a corporation which paid tens of millions of dollars to have a redundant network is now operating with a single point of failure. Brutal.


----------



## gardner (Feb 13, 2014)

My Petrocan phone is still not sending texts, so it's not all better yet.


----------



## Beaver101 (Nov 14, 2011)

jargey3000 said:


> back!
> what'd I miss?!


 .. nothing. Who's rogers?


----------



## andrewf (Mar 1, 2010)

My employer is ticking along just fine. Why? We have redundant connections. It is completely unacceptable for this to have happened, twice now, in a couple years. 

Now, we had a major IT failure last December, but it was not something very noticeable to the public or media. It took over a week to stabilize operations but we largely were able to keep operating. I have a lot of sympathy for the many people at Rogers who are having a really terrible day.


----------



## milhouse (Nov 16, 2016)

Tostig said:


> I'm sort of curious how the employees of Rogers actually discovered or concluded that there was a major outage.
> 
> Any Rogers employee who were able to be in contact with each other were obviously NOT on a Rogers phone, internet or cell phone. The CEO or other executives must have learned about it on the radio, like CBC and then discussed it when they were all in the same office together because conferences would not have been possible either.


That's a pretty good question.  
You'd think Rogers staff working from home wouldn't have the ability to log in or contact other staff because they'd all be on Rogers home internet, home phone, and cell service and would start asking themselves wtf is happening in isolation. Roger's VP spokesperson being interviewed on CBC indicated the problem is with Roger's core network. I'm curious how much of their internal corporate network and systems rely on their core network. On the otherhand, I'm assuming they would be able to isolate network segments so they could partially function internally.


----------



## milhouse (Nov 16, 2016)

Everything points to a software / firmware problem likely due to a patch or upgrade. Whereas, if it was a hardware problem, you'd assume it would be easy to swap the bad device out. However, the length of the outage really is ridiculous. 

They must have redundancy but what happens if you use the same devices on the main and backup routes and it's the software on the devices that's causing the problem? I suppose you can use devices from different vendors for each route but think about the complexity, effort, and cost of testing each different combination of the devices and routes every time you install a patch or do an upgrade, etc.
For the consumers/businesses, the way to go for redundancy is to have your wired service from one provider and your wireless service from another provider, though you likely lose savings from bundling.


----------



## ian (Jun 18, 2016)

I worked for a vendor and saw everything over the years.

From a stock brokerage house that did a three way backup and DR program (with regular review and practice) to a large hospital that went down for three days simply because of extremely sloppy IT practices and a non existent DR plan.

It can be shocking. So much so that we would send letters, etc to the senior exec team when we came across this type of incompetence and risk.


----------



## The TaxMan (Apr 28, 2021)

Fido user here. Nothing new, we went through this same thing about a year ago (or longer). People all over Canada not able to even log into their broker accounts, or CRA, no text messaging, etc, etc, etc. IIRC all Fido did for Canadians across the country (after lying for 24 hours that there was no outages), was offer something like a few free minutes for their troubles, when people had resulted in life-changing losses.

Yesterday, at my work even, employees could not punch-in their time-clock. None of our cash registers could process debit/credit cards the whole day, etc. At first I wondered if the Russians were carring out their cyber-attack on the West as they had threatended. But... as I said, we've been through this show before.

God help us if Russia really does make-due on its promise. We have lost.


----------



## MrMatt (Dec 21, 2011)

milhouse said:


> Everything points to a software / firmware problem likely due to a patch or upgrade.


Reports suggest there was an issue with the routing tables (ie traffic control maps for the data). 
The issue is that if you issue a routing update, it gets widely propagated.
Rogers really should do something to at least partially segment their network.
Of course it's WAY more complicated than that.


----------



## MrsPartridge (May 15, 2016)

I hope this ends the whole Rogers buys Shaw deal. What we experienced on Friday should send a message to the CRTC to kill that deal.


----------



## sags (May 15, 2010)

A national blackout of services is rare, but local or widespread Provincial blackouts are quite common with Rogers.

It has gotten to the point where people say if the wind blows or a few raindrops fall..........Rogers service will go out.

In our neighborhood, it isn't uncommon to see rogers cable strung from the boxes in the park behind us to individual townhouses for months at a time.

Unfortunately, there isn't a lot of quality competition in Canada.


----------



## MrMatt (Dec 21, 2011)

sags said:


> A national blackout of services is rare, but local or widespread Provincial blackouts are quite common with Rogers.
> 
> It has gotten to the point where people say if the wind blows or a few raindrops fall..........Rogers service will go out.
> 
> ...


Same for Bell, which is actually much worse in several parts of my neighbourhood. It really matters what block you're on.


----------



## m3s (Apr 3, 2010)

Yea I was going to sign up with Fizz mobile which uses Rogers/Videotron network and lets you carry over or gift unused data

Lots of comments online about known network issues with Rogers that don't seem to get addressed by Rogers. It would be ok to have some network issues but sounds like they just aren't repairing known deadspots that happened after storms etc

Bell is known to be the most expensive but their less known pre-paid plans look ok. There's also Koodo and Public which use the Bell/Telus network. Public was better before Telus bought it


----------



## Juggernaut92 (Aug 9, 2020)

I worked at a health cente in the past and they used rogers fiber as their main internet connection. They used rogers cable as their backup internet. Sometimes when the Fibre went down the cable went down as well even though initially rogers said those two services were separated at some point.

After knowing that and the issue we had yesterday and last year with rogers I am going to be a bit more cautious going forward at work and at home with rogers service. Will probably switch to telus for cell phone pla after contract with fido is done.


----------



## Tostig (Nov 18, 2020)

MrsPartridge said:


> I hope this ends the whole Rogers buys Shaw deal. What we experienced on Friday should send a message to the CRTC to kill that deal.


Maybe not. Perhaps the acquisition of Shaw would mean that future Rogers outages would kick in Shaw services as backup.


----------



## andrewf (Mar 1, 2010)

My home internet service was only restored around 6 pm today. 38hr outage!


----------



## kcowan (Jul 1, 2010)

We have been here in Vancouver so long that Shaw took over from Rogers and our home internet improved substantially. Now we use Shaw Internet and Bell/Shaw mobility, with a Telus landline for good measure. I have Mexican Telcel as backup Internet.


----------



## james4beach (Nov 15, 2012)

andrewf said:


> My home internet service was only restored around 6 pm today. 38hr outage!


That's unbelievable. What is, this, 1997 ?


----------



## Ponderling (Mar 1, 2013)

Even after my signal came back about 25 hours later, inet searches were still not working right. 

I am a Globe and Mail subscriber, and use a tablet feature, Globe To Go, as a path to get a print out of the puzzles. 

On Rogers, G2G would not resolve. While site was still up I disconnected from my Rogers wifi, and hopped to my DSL ISP wifi signal. I have kept it alive while Rogers slowly works the bugs out of my now 3 month old service. 

Well hey, guys, guess what? Primus would resolve G2G no problem, so it was not a content problem, it ws a Roger search problem. 

For a while earlier in the day it would not resolve searches to bring up the Toronto Star either, something it does from time to time at my location So all is not sweet and well in the land of Rogers yet. 

The fact that it took them 6 weeks to diagnose that signal at the pedestal on the street was under spec for amplitude, and had been for the whole time I was hooked up should tell you volumes.

The in house Rogers router still does not like to let the our HP wired network printer talk back to the print driver on newer computers is still ongoing.


----------



## james4beach (Nov 15, 2012)

I think the Rogers incident shows the appeal of having two carriers. For example, if your home internet is with Rogers, then you'd want to have a cell phone on a different network. You could even tether your phone to your laptop or desktop computer (using your phone's Wifi hot-spot) giving you connectivity during an outage.

I also keep an extra cell phone (charged, in my Emergency kit) with pay-as-you-go, on another network. This is in case of some natural disaster, power outage, earthquake etc, in case I lose my primary phone's service. For this, I'm using Speakout from 7/11 which happens to be on Rogers, but that's my only Rogers service.

I'm also really hoping that Rogers is not allowed to buy Shaw. We need diverse carriers in Canada, not concentration among 2 or 3.



Ponderling said:


> For a while earlier in the day it would not resolve searches


I have found ISP's domain name servers (DNS) to be unreliable. For many years now, I've always set my DNS servers to some external ones that I trust.

Examples would be... and see a larger list of free services here
1.1.1.1
1.0.0.1

I personally really like the Cloduflare DNS. They also have a set of servers which apply malware filtering, described here
1.1.1.2
1.0.0.2


----------



## james4beach (Nov 15, 2012)

Also, it seems much of the Interac system, and debit payments, were down due to the Rogers outage.

Does Interac seriously not have redundancy to multiple carriers? What kind of amateur hour is this?

I was out shopping, and saw some customers turned away (couldn't buy anything) because debit cards were not working.


----------



## like_to_retire (Oct 9, 2016)

james4beach said:


> I also keep an extra cell phone (charged, in my Emergency kit) with pay-as-you-go, on another network. This is in case of some natural disaster, power outage, earthquake etc, in case I lose my primary phone's service. For this, I'm using Speakout from 7/11 which happens to be on Rogers, but that's my only Rogers service.


haha, great minds. Yeah, that's identical to my situation. I have everything on Bell, so I keep an extra cell phone in my desk drawer that uses SpeakOut. I like SpeakOut since their re-up interval is a year. I decided to get this backup system after I cancelled my home phone some time ago and it left me with a single cell phone and provider - which just isn't safe.

ltr


----------



## cainvest (May 1, 2013)

james4beach said:


> I also keep an extra cell phone (charged, in my Emergency kit) with pay-as-you-go, on another network. This is in case of some natural disaster, power outage, earthquake etc, in case I lose my primary phone's service.


Apparently phones without a SIM card can still make 911 calls. If you have an old phone around you can use that for emergencies or use some local free Wifi.


----------



## james4beach (Nov 15, 2012)

cainvest said:


> Apparently phones without a SIM card can still make 911 calls. If you have an old phone around you can use that for emergencies or use some local free Wifi.


That's really interesting, thanks. I did not know. Are you sure 911 works without a SIM card?

Or do you need a SIM card in there, even if it's inactive / expired?

It makes me wonder which network the 911 call goes over if there isn't a SIM card in the phone. We heard that some people lost 911 service due to the Rogers outage, but what if they had taken the SIM card out and tried 911?


----------



## cainvest (May 1, 2013)

james4beach said:


> That's really interesting, thanks. I did not know. Are you sure 911 works without a SIM card?


Try it, just pull your SIM card ... it should say "emergency calls only" when you power it up.

Also -> Can I call 911 if my SIM card is not working? | Rogers

lol, bad example but Rogers was the first link to come up.


----------



## TomB16 (Jun 8, 2014)

Ponderling said:


> The in house Rogers router still does not like to let the our HP wired network printer talk back to the print driver on newer computers is still ongoing.


That isn't a problem with the Rogers network. Make sure the printer is plugged into a LAN port and not a WAN port. The printer needs to be inside your NAT cloud.


----------



## ian (Jun 18, 2016)

There are many customers who still do not have their cell and internet services restored as of late Sunday afternoon. Two day outage for these folks.


----------



## MrMatt (Dec 21, 2011)

TomB16 said:


> That isn't a problem with the Rogers network. Make sure the printer is plugged into a LAN port and not a WAN port. The printer needs to be inside your NAT cloud.


Some of the modem/routers don't behave well without the WAN connection. Even though it really shouldn't have an impact, my LAN wasn't routing well.


----------



## ian (Jun 18, 2016)

So glad that we are not Rogers customers. We suffered through their cell service in Vancouver for a few years. That was enough. Customer service.....what customer service?


----------



## Beaver101 (Nov 14, 2011)

$6 compensation for "2 days" downtime for small businesses. WOWIE. Lawsuits, bring them on.


----------



## Beaver101 (Nov 14, 2011)

ian said:


> So glad that we are not Rogers customers. We suffered through their cell service in Vancouver for a few years. That was enough. Customer service.....what customer service?


 ... that's because they don't know the definition of a "customer", never mind about "service".

The attitude(s) from them Rogers' executives is that "you're lucky you got us Rogers to give you an alternative."

As for Canada's Communications MP's suggestion as noble as it is that all the networks "help" each other during outages, if other service providers go or gets bogged down due one rogue provider, then as a "customer", I would want my money back for the bog-down or lack of service thereof regardless of who my provider is then. 

I think he should be suggesting the providers be coughing up a Customer's Bill of Rights for Internet/mobile Services and/or amending to strengthen it. And then ENFORCING IT.


----------



## Numbersman61 (Jan 26, 2015)

A big shout out to Ted Rogers. It was a smart move in getting rid of the CEO whose entire career was in telecommunications.


----------



## milhouse (Nov 16, 2016)

FP article saying Interac is adding another supplier in addition to Rogers to provide redundancy.









Interac to bolster payment network with back up supplier after Rogers outage


Millions of Canadians locked out of online payments last Friday after telco suffered national service disruption. Read more here




financialpost.com


----------



## kcowan (Jul 1, 2010)

ian said:


> So glad that we are not Rogers customers. We suffered through their cell service in Vancouver for a few years. That was enough. Customer service.....what customer service?


Yes they disappeared shortly after they closed their video/cellular shops.


----------



## kcowan (Jul 1, 2010)

I am surprised that Interac would depend on any single supplier.


----------



## kcowan (Jul 1, 2010)

We had an outage to our primary datacenter when the fibre optic trunk got severed by a backhoe. We had paid Bell for a separate redundant trunk for years, So we got a credit for several years and relieved them of a potential lawsuit for fraud. 

Fortunately, we had redundant links through the prairie and BC datacentres but with substantially reduced performance!


----------



## TomB16 (Jun 8, 2014)

I just watched the CEO Message to customers on CTV News.

There must be some nuance not captured in Mr. Staffieri's status brief. He makes it sound like they downloaded a firmware update and rolled it wide without testing. My gawd....

I've come to know some of the engineers on the Bell side of the world and those guys are super sharp. Total professionals. The reason the Bell network is so stable is because it is operated by many outstanding people. I only know a couple of people from Rogers and not high level so I should not judge but I'm starting to think some of the Bell guys should raise some cash, buy Rogers, and turn it into a professional shop.


----------



## Covariance (Oct 20, 2020)

TomB16 said:


> I just watched the CEO Message to customers on CTV News.
> 
> There must be some nuance not captured in Mr. Staffieri's status brief. He makes it sound like they downloaded a firmware update and rolled it wide without testing. My gawd....
> 
> I've come to know some of the engineers on the Bell side of the world and those guys are super sharp. Total professionals. The reason the Bell network is so stable is because it is operated by many outstanding people. I only know a couple of people from Rogers and not high level so I should not judge but I'm starting to think some of the Bell guys should raise some cash, buy Rogers, and turn it into a professional shop.


Best practice is to only deploy into production if there is a restoration point to roll back to in the event of a problem. Some times a restoration point is not possible. For instance a major software or firmware upgrade. In these situations the only guarantee is to have duplicate equipment. Cut to new equipment with the new software while the old is still in place to fall back to if there is an issue. This becomes a budget issue. A lot of extra capital equipment required for one day.


----------



## Beaver101 (Nov 14, 2011)

TomB16 said:


> I just watched the CEO Message to customers on CTV News.
> 
> There must be some nuance not captured in Mr. Staffieri's status brief. He makes it sound like they downloaded a firmware update and rolled it wide without testing. My gawd....


 ... never saw that video message but then I'm not surprised at all given the attitude(s) of its executives are "you, the users (note, didn't use the word "customer" here) are the problem." 

Moreover, wouldn't be surprised they hired a bunch of IT wannabees or IT course dropouts to handle that "upgrade". And it wouldn't surprise me that "some bodies" are gonna to be sacked for that financial hit that the "executives" took - never mind "Rogers" took.



> I've come to know some of the engineers on the Bell side of the world and those guys are super sharp. Total professionals. The reason the Bell network is so stable is because it is operated by many outstanding people. I only know a couple of people from Rogers and not high level so I should not judge but I'm starting to think some of the Bell guys should raise some cash, buy Rogers, and turn it into a professional shop.


 ... see above. Cheap outs get cheap-ins. But then Rogers have to save as much dinaros to pay its executives its bonuses, comps, etc. to buy its personal toys or that Shaw deal (scary for its customers going forward)>

Any idiot not in IT knows for any upgrade, you test it on a secondary, non-live platform first. Oh well.


----------



## milhouse (Nov 16, 2016)

Beaver101 said:


> Moreover, wouldn't be surprised they hired a bunch of IT wannabees or IT course dropouts to handle that "upgrade". And it wouldn't surprise me that "some bodies" are gonna to be sacked for that financial hit that the "executives" took - never mind "Rogers" took.


Yeah, someone or a few people are going to end up losing their jobs over this. It's one of the worst combinations of financial and reputational damage. 



Beaver101 said:


> Any idiot not in IT knows for any upgrade, you test it on a secondary, non-live platform first. Oh well.


Got to say it's also about good organizational change management/oversight practices too. The techs just want to get the change implemented and there needs to be good oversight by the change management team asking the right questions about the risks, potential impacts, and recovery strategies.


----------



## Covariance (Oct 20, 2020)

milhouse said:


> Yeah, someone or a few people are going to end up losing their jobs over this. It's one of the worst combinations of financial and reputational damage.
> 
> 
> Got to say it's also about good organizational change management/oversight practices too. The techs just want to get the change implemented and there needs to be good oversight by the change management team asking the right questions about the risks, potential impacts, and recovery strategies.


Also possible "the techs" wanted redundant equipment in case it didn't work and the execs wouldn't fund it. 

Unless someone knows for sure and is prepared to tell all.


----------



## milhouse (Nov 16, 2016)

Covariance said:


> Also possible "the techs" wanted redundant equipment in case it didn't work and the execs wouldn't fund it.
> 
> Unless someone knows for sure and is prepared to tell all.


Yep. We're all kind of just guessing.


----------



## MrMatt (Dec 21, 2011)

TomB16 said:


> I just watched the CEO Message to customers on CTV News.
> 
> There must be some nuance not captured in Mr. Staffieri's status brief. He makes it sound like they downloaded a firmware update and rolled it wide without testing. My gawd....
> 
> I've come to know some of the engineers on the Bell side of the world and those guys are super sharp. Total professionals. The reason the Bell network is so stable is because it is operated by many outstanding people. I only know a couple of people from Rogers and not high level so I should not judge but I'm starting to think some of the Bell guys should raise some cash, buy Rogers, and turn it into a professional shop.


The thing is that they should have done testing, then partial rollouts.
To live rollout the entire network all at once is just crazy these days.


----------



## Covariance (Oct 20, 2020)

MrMatt said:


> The thing is that they should have done testing, then partial rollouts.
> To live rollout the entire network all at once is just crazy these days.


Sometimes you need to do the equivalent of open heart surgery. A major upgrade on core tech. No way around it. SO it's all about risk mitigation.


----------



## Retired Peasant (Apr 22, 2013)

The wee hours on a Sunday morning would have been a better choice.


----------



## MrMatt (Dec 21, 2011)

Covariance said:


> Sometimes you need to do the equivalent of open heart surgery. A major upgrade on core tech. No way around it. SO it's all about risk mitigation.


I honestly can't imagine such a case.
If they segmented their network, they should be able to upgrade parts at a time.

They absolutely had to upgrade their entire national infrastructure, and couldn't have possibly done it in phases? I simply don't believe it.


----------



## cainvest (May 1, 2013)

MrMatt said:


> They absolutely had to upgrade their entire national infrastructure, and couldn't have possibly done it in phases? I simply don't believe it.


You'll never know unless you are on the inside and take part in the right meetings.


----------



## MrMatt (Dec 21, 2011)

cainvest said:


> You'll never know unless you are on the inside and take part in the right meetings.


The reason that the Bell Network didn't go down is because it was properly isolated from the Rogers network.
Rogers could (and should) do the same thing internally.


----------



## cainvest (May 1, 2013)

MrMatt said:


> The reason that the Bell Network didn't go down is because it was properly isolated from the Rogers network.


Bell is a different network, why would they go down?


----------



## sags (May 15, 2010)

We received another email sayin blah blah blah and an increase in the rebate.

What it doesn’t say is they will keep customers informed when there are any problems in the future …..something they have never done in the past.

They wouldn’t even answer queries from local radio media so people could be informed of the network interruptions.

We end up asking neighbours if they have service.


----------



## sags (May 15, 2010)

I am not convinced Bell has that solid a system either. On June 16 and 17 they sent out tweets that there would be outages in Ontario and Quebec due to bad weather.

Private industry is good for some things but critical infrastructure isn’t one of them.


----------



## james4beach (Nov 15, 2012)

It's kind of funny that I'm getting such good mobile service in Canada, since I stuck with my American telco (T-Mobile).

If any of you have an American address and can get on T-mobile I highly recommend it, especially if you're a world traveller. You'll get free data roaming anywhere on earth. I'm on a Canada-friendly plan where calls are free/unlimited in Canada & US.

But what happens in Canada is that my US phone hops between Canadian networks. I've seen it on Telus, Rogers, Bell depending on the city. Currently I'm on Telus but I'm pretty sure I was on Bell just a week ago.

Thankfully, 911 also works. A few months ago I was nearly run off the road by an aggressive driver / road rage, and I had to flee from the other car, and called 911. Was routed to the 911 dispatch for the city I was in. I don't know what the dispatchers see when a US phone calls into 911 but it seemed to work flawlessly as far as I could tell.


----------



## TomB16 (Jun 8, 2014)

The other driver was probably enraged by your denigration of his corporate distributions.


----------



## MrMatt (Dec 21, 2011)

cainvest said:


> Bell is a different network, why would they go down?


They wouldn't because they're segregated.

Why does Rogers have a single nationwide network? Why didn't they break it into several smaller easier to administer units?
A single large network is simpler and cheaper in some ways, but several smaller networks has benefits, one being that a single upgrade shouldn't wipe out the whole network, at most only that part.

Realistically there is no reason for my cable modem to be on the same logical and administrative network as that of someone on the other side of the country, or a commercial business on yet another side. It's a bad design, it should be pretty much impossible for a single action, of any type anywhere on the network to bring down almost the whole company..

That's exactly why new technologies (web3 & crypto) are so interesting, they're removing those single (or multiple) points of failure.


----------



## cainvest (May 1, 2013)

MrMatt said:


> They wouldn't because they're segregated.
> 
> Why does Rogers have a single nationwide network? Why didn't they break it into several smaller easier to administer units?
> A single large network is simpler and cheaper in some ways, but several smaller networks has benefits, one being that a single upgrade shouldn't wipe out the whole network, at most only that part.


Not sure where you're getting this from .... did someone leak hardware/software topology maps of Bell vs Rogers onto the internet? 

In other words, could Rogers have done it in sections ... my guess is most likely yes.


----------



## MrMatt (Dec 21, 2011)

cainvest said:


> Not sure where you're getting this from .... did someone leak hardware/software topology maps of Bell vs Rogers onto the internet?
> 
> In other words, could Rogers have done it in sections ... my guess is most likely yes.


If they intentionally broke multiple sections at the same time, or sequentially broke one and then moved on to the next one to break it suggests a level of incompetence beyond what I think even Rogers is capable of.

more likely they had an improperly segregated network and just tried to "upgrade everything" at once. Which is colossally stupid.


----------



## cainvest (May 1, 2013)

MrMatt said:


> more likely they had an improperly segregated network and just tried to "upgrade everything" at once. Which is colossally stupid.


Like I said before, you'll likely never know what the fault was or where to assign blame. There is no way to "armchair quarterback" the possible issue without knowing the details. Of course if you enjoy pure speculation, which many do here, please continue.


----------



## MrMatt (Dec 21, 2011)

cainvest said:


> Like I said before, you'll likely never know what the fault was or where to assign blame. There is no way to "armchair quarterback" the possible issue without knowing the details. Of course if you enjoy pure speculation, which many do here, please continue.


No, but anyone who has designed important systems of any type knows to separate them as much as is reasonable.


----------



## cainvest (May 1, 2013)

MrMatt said:


> No, but anyone who has designed important systems of any type knows to separate them as much as is reasonable.


Who knows ... but you might want to check if they get a new CTO in the coming months.


----------



## TomB16 (Jun 8, 2014)

The problem domain for this event is far from infinite. Internet industry professionals can narrow this event down to a handful of scenarios with only two being all that likely.

The entirety of a national network went down, both voice and data, so this was not a case where one more degree of redundancy at one location would have helped.

As for the bad firmware idea, this also seems a highly unlikely corner case. I would not directly blame this on Nokia but there is no doubt the Nokia tool suite is a sharp knife with which many operators have disembowelled themselves.


----------



## milhouse (Nov 16, 2016)

james4beach said:


> It's kind of funny that I'm getting such good mobile service in Canada, since I stuck with my American telco (T-Mobile).
> 
> If any of you have an American address and can get on T-mobile I highly recommend it, especially if you're a world traveller. You'll get free data roaming anywhere on earth. I'm on a Canada-friendly plan where calls are free/unlimited in Canada & US.


How long have you've been using the Tmobile plan primarily in Canada? I was under the (incorrect?) impression that if they noticed your usage was primarily in Canada instead of the US that they would kick you off.


----------



## TomB16 (Jun 8, 2014)

IMO, the most interesting aspect of this crap fountain is the following:


Rogers has been the least reliable national network for many years
They long ago promised to improve their reliability but this does not seem to be happening

It is extremely rare to turn a poorly running business around. And yet, legions of investors buy distressed companies that have made a litany of bad decisions thinking their fortunes will change. Business fortunes rarely change, particularly without major staff changes. While it does happen, it requires massive effort and ruthless resolve to turn things around.


----------



## james4beach (Nov 15, 2012)

milhouse said:


> How long have you've been using the Tmobile plan primarily in Canada? I was under the (incorrect?) impression that if they noticed your usage was primarily in Canada instead of the US that they would kick you off.


I also thought they would terminate my service. I've been continuously in Canada for over 2 years, and don't think I've ever been on a US cell tower for that time.

I won't be surprised if they kick me off. Maybe they made an exception during the pandemic.


----------



## kcowan (Jul 1, 2010)

james4beach said:


> I won't be surprised if they kick me off. Maybe they made an exception during the pandemic.


AT&T offered that service in Mexico and it was two years before they had to introduce limits on foreign roaming, I recall the cap was 25% of total usage but with a further condition on failing to operate in Mexico for more than 3 months in a row. But it was only volume that caused them to change.


----------



## james4beach (Nov 15, 2012)

kcowan said:


> AT&T offered that service in Mexico and it was two years before they had to introduce limits on foreign roaming, I recall the cap was 25% of total usage but with a further condition on failing to operate in Mexico for more than 3 months in a row. But it was only volume that caused them to change.


Interesting, thanks. I am now back to occasional US travel, so my phone gets back onto their primary US network once in a while.

But I'm still in Canada nearly all the time so it will be interesting to see if they pull the plug on me.

I had no idea how long the pandemic would be restricting travel, so I didn't want to give up the plan a couple years ago.


----------



## MrMatt (Dec 21, 2011)

MrMatt said:


> Reports suggest there was an issue with the routing tables (ie traffic control maps for the data).
> The issue is that if you issue a routing update, it gets widely propagated.
> Rogers really should do something to at least partially segment their network.
> Of course it's WAY more complicated than that.


FYI, this was supported by Cloudflare who understands networking.








Cloudflare’s view of the Rogers Communications outage in Canada


An outage at one of the largest ISPs in Canada, Rogers Communications, started earlier today, July 8, 2022, and is ongoing (eight hours and counting), and is impacting businesses and consumers.




blog.cloudflare.com







cainvest said:


> MrMatt said:
> 
> 
> > No, but anyone who has designed important systems of any type knows to separate them as much as is reasonable.
> ...


You're not understanding what I'm saying with Rogers vs Bell, I'm not saying one is better than the other. I'm just saying that if you PROPERLY segregate networks, the failure is contained only to that network. Rogers should not have built their whole network as a single system. IT should have been broken into segments.

Any serious system has distinct systems, in safety applications it is essential to separate out different functions to ensure that failure in one isn't catastrophic. This is basic engineering practice.

My point is that they should have done it this way, and now they will go, or at least claim to.
Honestly they should break it down much smaller to prevent such massive and widespread issues.
*Rogers to separate wireless, wireline traffic to prevent outages like July 8th*








Rogers to separate wireless, wireline traffic to prevent outages like July 8th


MobileSyrup has learned that Rogers plans to separate its wireless and wireline traffic to avoid another major outage like July 8th.




mobilesyrup.com


----------



## cainvest (May 1, 2013)

MrMatt said:


> You're not understanding what I'm saying with Rogers vs Bell, I'm not saying one is better than the other. I'm just saying that if you PROPERLY segregate networks, the failure is contained only to that network. Rogers should not have built their whole network as a single system. IT should have been broken into segments.


Makes more sense now with the mobilesyrup link provided.

Also doesn't make sense they didn't test the update to their BGP routers AND that they didn't have a fast way to restore them if a problem came up.


----------



## MrMatt (Dec 21, 2011)

cainvest said:


> Makes more sense now with the mobilesyrup link provided.
> 
> Also doesn't make sense they didn't test the update to their BGP routers AND that they didn't have a fast way to restore them if a problem came up.


For such important things they should have multiple steps and safeguards to make this type of change. this really was amateur hour.


----------



## Beaver101 (Nov 14, 2011)

MrMatt said:


> For such important things they should have multiple steps and safeguards to make this type of change. this really was *amateur hour.*


 ... I bet its shareholders are happy with the savings there to go to the share price and dividends payout ...  LMAO.


----------



## Gator13 (Jan 5, 2020)

Rogers CTO resigned today.


----------



## cainvest (May 1, 2013)

Gator13 said:


> Rogers CTO resigned today.


Saw that one coming on the 14th.


----------



## milhouse (Nov 16, 2016)

Gator13 said:


> Rogers CTO resigned today.





cainvest said:


> Saw that one coming on the 14th.


Yeah, someone was going to have to take the hit. 

Q2 Earnings call next week should be interesting.









Rogers replaces technology chief after massive outage draws fire


Unprecedented outage shut banking, transport and government access for millions




financialpost.com


----------



## Beaver101 (Nov 14, 2011)

cainvest said:


> Saw that one coming on the 14th.


 ... want to make that earlier like 12th of July, 2022?


----------



## cainvest (May 1, 2013)

Beaver101 said:


> ... want to make that earlier like 12th of July, 2022?


No, the 14th is when I posted ...


cainvest said:


> Who knows ... but you might want to check if they get a new CTO in the coming months.


----------



## Beaver101 (Nov 14, 2011)

Gator13 said:


> Rogers CTO resigned today.


 ... are you sure he "resigned" and not fired? You know how "nice" Rogers' hench executives are. Some says it's a "dream" company to work for ... LMFAO ...


----------



## Beaver101 (Nov 14, 2011)

cainvest said:


> No, the 14th is when I posted ...


 ... okay, you're going by the "date" "you" posted. I'm going by the aftermath date.


----------



## james4beach (Nov 15, 2012)

This is pretty funny. Apparently, Rogers employees could not connect to any of the Rogers systems to fix the problem, because their own connections (devices) were down. Only Rogers employees using the competition's networks --- like Bell --- were able to work on the problem.

*Total amateur hour*. In my consulting work, I advise companies to prepare back-up connection methods in case of outages (like internet attacks), suggesting they keep a cell phone on a different network, or even a second modem on a different service.

At home, I keep a spare cell phone in my emergency kit for the same reason -- backup connectivity in case my primary network goes out.



https://www.cbc.ca/news/business/rogers-letter-outage-crtc-1.6530067


----------



## Money172375 (Jun 29, 2018)

james4beach said:


> This is pretty funny. Apparently, Rogers employees could not connect to any of the Rogers systems to fix the problem, because their own connections (devices) were down. Only Rogers employees using the competition's networks --- like Bell --- were able to work on the problem.
> 
> *Total amateur hour*. In my consulting work, I advise companies to prepare back-up connection methods in case of outages (like internet attacks), suggesting they keep a cell phone on a different network, or even a second modem on a different service.
> 
> ...


I remember when the bank’s phones were switched to voip. They gave each branch one old phone and hooked it up to the old copper line in the server room. Was our only means of communication if the internet or power went out.


----------



## Beaver101 (Nov 14, 2011)

^^ It wouldn't surprised me that its IT disaster recovery team is outsourced.

Please call back in 12 hours during "business" (meaning holidays and weekends are excluded) hours ... and please try again ... and again ... and again ... YAWN ... LMAO.


----------



## Beaver101 (Nov 14, 2011)

Just after I posted above, this updated newspiece says:

Rogers says it couldn't have restored emergency services any faster during outage



> _The Canadian Press Published Friday, July 22, 2022 10:57PM EDT
> 
> TORONTO -- Rogers Communications Inc. says in a submission to the federal telecom regulator that it could not have restored emergency services any faster during a widespread service outage earlier this month that crippled the company's networks and affected millions of customers across Canada.
> 
> ...


 ... WOW, just amazing "response" re the bolded part " ..._ *tackle the problem* *at its source* rather than _... ". 

I say the first and foremost "source" of the problem(s) is at Rogers' Executive Suite. 

And I wonder if there're any mirrors in them restrooms over at its headquarters.


----------



## sags (May 15, 2010)

I would be happy if Rogers would stop all the robocalls and texts coming from offshore but showing up as a local phone number.

I am getting all kinds of them lately.....on both the home phone and cellphone. Everything from CRA scams, to the latest ......._.you have a missed delivery _scams.

Add in the carpet cleaners, Visa account scams, and all the rest and it is becoming a big problem for a lot of people.


----------



## ian (Jun 18, 2016)

I cannot imagine why anyone would consider Rogers as a service provider or remain a customer. Two outages within a year. Sub standard customer service. Not to mention their cavalier attitude post the second outage.

Why would anyone deal with a provider, or with a provider who used Rogers infrastructure, unless there was no other choice.

We had substandard cable/internet service from Shaw a number of years ago and poor customer service.

It was not until we punted them and experienced an alternate service provider that we realized how poor they were....at least in our area.


----------



## Beaver101 (Nov 14, 2011)

Rogers pursuing deal with competitors to juggle 911 calls in case of outage


> _
> Published Sunday, July 24, 2022 6:01PM EDT
> TORONTO - Rogers Communications Inc. says it's working to ensure emergency calls always go through and no future outage will take down both cellular and internet services, as it continues damage control following a blanket network outage earlier this month.
> 
> ...


 ... and it's gonna to take them "3 years" to ensure this???? HOLY CRXP.


----------



## Beaver101 (Nov 14, 2011)

Update Monday July 25, 2022:

House of Commons committee will hold two hearings on Rogers outage today



> ... _The committee will seek answers about the cause of the outage, its overall effect, and best practices to avoid future outages and better communicate with the public during such emergencies. ... _


 ... I would really be interested in hearing the answers if not results of this enquiry. Don't have high hopes of anything accomplished other than something to do with a free-lunch there.... for the participants that is.


----------



## Beaver101 (Nov 14, 2011)

How a coding error caused Rogers outage that left millions without service

Why in the hell do Roger "customers" need to know such details like "coding errors ... etc.". Are they expecting its customers to do its IT's jobs? Like how to call 911 when your service with Rogers is down. Yeah, go to the nearest coffee shop and call there ... provided it doesn't use Rogers. Or better yet, find a Bell telephone booth there whilst you're laying on the floor trying to reach your phone (any version).

Interesting enough its "CTO" was in Portugal with this "systems upgrade " ... I wonder where was its senior VP of "Customer Service" (if it exists even) at that time? On the phone with the CEO I bet ... coming up with some PR BS.


----------



## Covariance (Oct 20, 2020)

Redacted report is available on the CRTC website. Summary - they had core routers from two different vendors that were different (incompatible) in how they managed BGP routing tables.


----------



## Tostig (Nov 18, 2020)

Covariance said:


> - they had core routers from two different vendors that were different (incompatible) in how they managed BGP routing tables.


"Tell me this isn't a government operation." Gene Kranz

It's not. It's a private-sector operation.


----------



## Beaver101 (Nov 14, 2011)

‘Unprecedented incident’ took down ‘brain’ of network, Rogers execs tell MPs



> By Christine Dobby, Toronto Star, Mon., July 25, 2022
> Article was updated 30 mins ago
> 
> ...
> ...


 ... another way of saying Rogers' users (and then Shaw) should be prepared for rates increase and pay up for continued wonky usage. In the meantime, I wonder how much are those lawsuits gonna cost? 

and then,



> ...
> 
> _Innovation Minister François-Philippe Champagne, who was first up at the hearing Monday morning at 11 a.m., told MPs that he was the one who first reached out to Staffieri during the outage to find out what was happening.
> 
> ...


 ... good luck to Mr. Champagne's attempt to dispense "advice" aka "hot air" to Rogers.


----------



## ian (Jun 18, 2016)

I saw a clip of Rogers CIO and CIO making reference in a meeting with MP's about Rogers being Canada's most reliable network 

Don't they know it is not good to snow the snowmen? Let alone joe public. 

They embarrassed themselves.


----------



## Beaver101 (Nov 14, 2011)

ian said:


> I saw a clip of Rogers CIO and CIO making reference in a meeting with MP's about Rogers being Canada's most reliable network
> 
> Don't they know it is not good to snow the snowmen? Let alone joe public.
> 
> They embarrassed themselves.


 ... they don't know what's shame is, let alone embarassment. Let it snow, let it snow ... lawsuits.


----------



## Beaver101 (Nov 14, 2011)

Rogers earns $409 million in Q2 as wireless service revenue soars 11 per cent

Like I said before, shareholders are happy. But what happens if you happen to be a shareholder, AND a subscriber/user? Laugh and cry at the same time? Talk about shooting oneself in both feet ...


----------



## Covariance (Oct 20, 2020)

Companies have no soul. They are focused on their own interest. While it could be argued that providing a high quality reliable service is in their best interest, ultimately a financial trade off places a limit on it. I place "the blame" squarely on Gov't that has allowed them to buy up all their competitors, reduce competition, and by extension remove redundancy and competitive alternatives. The anemic response is similar to the promise to get them to reduce the cost of cell phone service.


----------



## Beaver101 (Nov 14, 2011)

^ I say good luck with that ... more competition,service costs reductions, etc. as I hope you realize that both players are in bed with one another whilst the "customers" silently watches. Nothing new.


----------



## Beaver101 (Nov 14, 2011)

How should you be compensated in the event of a similar Rogers debacle? Some say it shouldn’t be up to your provider to decide



> _By Christine Dobby Business Reporter, Toronto Star, Wed., July 27, 2022
> 
> It shouldn’t be up to your telecom provider to decide how to compensate you if you’re left in the dark during network outages, academics and consumer advocates say.
> 
> ...


 ... I think drafting a Bill of Rights for consumers/customers would a good start. Classaction has been proposed / to start in Quebec (link is within the article).


----------



## Covariance (Oct 20, 2020)

Beaver101 said:


> ^ I say good luck with that ... more competition,service costs reductions, etc. as I hope you realize that both players are in bed with one another whilst the "customers" silently watches. Nothing new.


Agreed. I only expect them to do what is in their best interest. And, I only expect elected Gov't to do what will get them re elected.


----------

