Failure Disrupts Phone System
in Los Angeles Area
This phone system failure took place in southern California on October 18,
2005. Not from an earthquake, terrorist attack or other disaster. It
was because of a software glitch.
Back to Alcorn EMA main page
Hello ... ? Glitch Kills Phones for Thousands, Failed software in Long Beach crimps cellphone, Internet and land-line service, including 911.
By Nancy Wride, James S. Granelli and Dan Weikel
Los Angeles Times Staff Writers
October 19, 2005
At least 150,000 customers and probably many more over a
large swath of Southern California lost their phone and Internet service for up
to 12 hours Tuesday because of a computer glitch at a Long Beach central
The outage, which also cut off 911 service from many land lines, struck
communities from Hermosa Beach to Newport Beach along the coast and as far
inland as Whittier and Garden Grove. Some cellphone service was also lost,
though on a more limited basis.
The problem, discovered at 2:23 a.m., was caused by a software error in a
downtown Long Beach office of Verizon Communications Inc. The malfunction
corrupted the main software that connects calls and operates the local network,
The backup system, which should have deployed, failed to activate, and Verizon
officials said they were trying to figure out why.
The hard-drive crash at the Long Beach facility Tuesday morning had a ripple
effect at 15 other Verizon central switching offices from Fountain Valley to
Redondo Beach and inland to Downey, as well as at dozens of cell towers
connected to the network.
The breakdown of telephone connections was the second time in about a month that
the failure of a small part of a complex network caused households across large
chunks of the region to lose a basic service.
In mid-September an electrical worker accidentally cut the wrong cable at a
power substation, shutting down power to much of Los Angeles for hours. Both
incidents underscore how basic utilities relied upon by millions of people are
vulnerable to relatively simple disruptions.
Even though the phone shutdown left some residents unable to reach 911 for
hours, authorities reported no known crime or medical problems resulting from
the telephone blackout.
But it did make for a vexing day across the coastal region.
On Main Street in Seal Beach, merchants had trouble because credit card scanners
didn't work. Surgeons at Long Beach Memorial Medical Center and Miller
Children's Hospital were able to perform scheduled operations only because their
cellphones would allow patients to reach them for post-op help. Several morning
flights at Long Beach Airport were delayed up to 45 minutes at the facility
because of phone-related problems with ground communications.
At Lowell Elementary School in Long Beach, it was strangely quiet in the front
office. "Only two calls made it through this morning, and our computer programs
aren't working, and we can't reach the district," secretary Betty Wohlgezogen
said at noon.
The state's fifth-largest city immediately had moved into what it declared
"communication failure protocol." At an emergency communication center near Long
Beach Airport from which Long Beach police and fire officials operated Tuesday,
a team of volunteer ham radio operators fanned out to 17 of the largest nursing
homes or assisted living centers to communicate with medics, hospitals and first
responders. The radio operator plan, city officials said, was designed to
prevent a repeat of the New Orleans hurricane tragedy in which several bedridden
patients were marooned and died at a nursing home.
The outage was so complete that Long Beach officials had to drive to KJZZ-FM
(88.1), the city's designated emergency broadcast station, to let the radio
station know what to tell listeners.
To cope with the failure of the 911 system, police departments in several
affected cities placed more patrol cars on the streets. .
Phone industry experts were surprised by the scope of the outage but said it
showed how interrelated the telecommunications network is — both landline phones
"People don't realize how interconnected telecom is," said Kathleen A. Dunleavy,
a spokeswoman for Sprint Nextel Corp. "People think wireless operates wholly
wirelessly, but it doesn't. The cell towers are dependent on [high-capacity] T-1
Cellphone service is typically wireless only to the nearest cell tower. From
there, high-capacity T-1 lines take the voice and data traffic, including
ordinary calls, e-mail and Internet browsing.
The 15 central offices hit by the outage are local hubs for telecommunications,
each one serving 10,000 to 15,000 customers, said Verizon spokesman Jonathan
Davies. The office contains switches, routers and other electronic gear to
connect calls and provide features such as call waiting and voicemail. It's
often where the long-distance lines and the Internet meet the local lines.
The company will analyze the cause and figure out ways to prevent another
Verizon Wireless, which is 55% owned by Verizon, has two backup networks
connecting its cell towers. One is a high-capacity fiber-optic network, the
other a wireless microwave system. Both of those cellular backup systems seemed
to have fewer problems than the land-line backup system in Long Beach that
Verizon and other carriers late Tuesday were still trying to figure out exactly
how widespread the outage was to both land and cellular phones.
T-Mobile USA said its customers also were affected by network congestion because
so many of them switched to cellphones to make calls. Customers as far south as
Laguna Beach reported problems, said spokesman Graham Crow.
Carriers typically have backup systems already in place, said analyst Charles
Golvin of Forrester Research Inc., a telecom research firm. Putting more in
wouldn't be economical, he said.
"Nothing is going to be foolproof," Golvin said.
Phone service was restored around 2 p.m. But the breakdown left both residents
and authorities with a new appreciation about how reliant they are on the phone
"The good lesson here is just how much we depend on our communications systems.
It's huge," said Laguna Beach Capt. Danell Adams. Under cloudy skies at the Seal
Beach Municipal Pier, Kent Trollen was philosophical about the
"It has been very peaceful," Trollen said. "But it accentuates the neurosis of
Back to Alcorn EMA main page