Earlier this month, Xcel Energy cut electricity to about 55,000 customers along the Front Range to reduce wildfire risks from wind gusts that topped 100 mph. Another 95,000 of its customers lost power because of downed power lines from those intense winds.
Those outages, planned or forced, removed power to about 40 cellular towers that Verizon Wireless relies on. Verizon had alternate power sources in place to power the towers, part of its larger game plan to keep its network going in extreme conditions
People could make calls or stream movies on their devices even if they couldn’t turn on the lights or watch television at home.
If Verizon, the nation’s second-largest wireless provider has its way, customers won’t ever experience a widespread network outage. And it goes further, maintaining a fleet of emergency equipment that can rush into disaster zones so cellular services can be restored or provided to emergency responders.
“Redundancy is the key,” said Blair Miller, a senior director of network assurance in Verizon’s Mountain Plains region, during a media tour on Tuesday.
In a facility tucked away in a nondescript industrial park in Aurora, Verizon employees monitor traffic on the network, changing weather conditions across the region, the status of hundreds of towers and other real-time feeds on large overhead screens.
The command room, staffed around the clock, is part of a larger switching and emergency response center, whose location Verizon keeps under wraps. The center and others like it around the country are vital to Verizon’s effort to keep its network operating under any scenario.
When a call, text or data request hits a cell phone tower from a mobile device, the signal usually moves into fiber optic lines. Locally, those signals flow into the company’s switching facility in Aurora.
Two big rooms, full of computer servers, each run at 40% to 50% capacity. If one of the two centers goes down, the traffic can flow into the other, said Jose Sanchez, a senior building engineer with Verizon.
Rows of computer servers, their flashing green lights, fill the role that human operators did in the early days of phone service when they manually pulled a line, said “please hold,” and patched a call into the right destination.
The switching equipment handles millions of packets of data at a time and redirects them at lightning speed. And while they never ask for a lunch break, they require constant power and relief from the loads of heat they release.
Thermal management is key to keeping any mobile switch center running. A dozen large A/C units, most waiting on standby, cool the switching centers. Three industrial-sized chiller units outside assist the units inside.
The switching equipment and heavy HVAC systems consume huge amounts of electricity. Keeping the power on at all times is key to ensuring the data packets don’t skip a beat and end up where they need to be.
Battery backups can power the two rooms for 10 to 13 hours if needed, Sanchez said. But the batteries are mainly there to handle the tiny gap in time that generators need to kick in.
It is only three seconds. But for data packets zooming around at 186,282 miles per second, a lot can go wrong in three seconds.
Four massive generators equipped with 12,000-gallon diesel fuel tanks stand ready outside. They switch on instantly when power is disrupted. Each one can run for between 11 to 13 days before needing to be refueled.
The Verizon center is on Xcel Energy’s priority list should chronic power shortages ever emerge. And if electricity were cut off completely and diesel shipments couldn’t arrive to keep the generators running, the generators could keep things going for close to a month, Sanchez said.
How people would charge their devices in that kind of environment — that’s an entirely separate question.
If that isn’t enough redundancy, cellular traffic in the region could be routed to another switching center, most likely Salt Lake City, assuming that location didn’t face a similar predicament.
Despite all those safeguards, the system isn’t foolproof. Verizon has built a fiber optic network along the Front Range and other areas with concentrations of customers, allowing it greater control. But in rural areas, it relies on third parties to haul its signals.
Failures in the fiber optic network can disrupt cellular service. That happened on Dec. 27-28 in 2018, when CenturyLink, now Lumen, suffered nationwide outages because of a bad network card in Denver.
As many as 22 million CenturyLink customers across 39 states were affected. About 17 million customers across 29 states lost reliable access to 911, according to a Federal Communications Commission study of the outage.
About 93,000 Verizon Wireless customers were impacted and about 1.9 million calls were blocked or lost in Montana and Wyoming, the FCC estimates.
No amount of redundancy in switching centers will help if the data isn’t arriving or arriving in an orderly way.
Verizon Frontline
Verizon’s Aurora location isn’t just about moving calls and data packets without downtime. It also serves as the base for the company’s local emergency response service — Verizon Frontline.
While the switching center is prepared for a large-scale catastrophe, localized emergencies are more common, like the state’s frequent wildfires, rarer floods, and the occasional rock slides, which in the case of Glenwood Canyon in 2021 cut fiber optic lines.
Verizon has a fleet of vehicles designed to restore or provide service in those situations, said Jared Hilzendeger, senior crisis response manager for Verizon’s Frontline team.
They include cellular on wheels or COWS, trailers that can be hauled in, connected to a fiber optic line and then broadcast a cellular or Wi-Fi signal. COWs, which have generators for power, are also used to provide extra capacity at big gatherings, like outdoor concerts.
Then there are COLTS, or cellular on light trucks. A COLT helped restore service in Boulder County when cellular towers burned up in the Marshal fire in late 2021 and stayed in place for two years assisting a new tower being built, Miller said.
If fiber isn’t available, equipment with satellite access can be brought in to connect to the internet that way.
In more difficult terrain where vehicles can’t be brought in, Verizone Frontline uses drones to broadcast a signal. Tethered drones with a connected power supply can stay airborne for long periods and were helpful in the Maui fires last year.
Verizon has developed robotic devices that can move into a dangerous area on their own, like a collapsed tunnel or toxic spill zone from a train derailment, Hilzendeger said. Those signals can be used to support monitoring equipment that can provide real-time information to responders.
“We do a lot of exercises with agencies and are always asking how our stuff compares with theirs,” Hilzendeger said.
