Verizon Service Disruption Wreaks Havoc at Kennedy Airport

[inanna]

Source: New York Times

MAY 30, 2016

A Verizon service disruption unleashed travel chaos at a terminal at Kennedy International Airport on Sunday night, grinding the check-in process to a near halt and forcing airline employees to hand-write boarding passes for thousands of unhappy passengers.

Large numbers of travelers were affected by the disruption, which began around 4 p.m. in Terminal 7, which is operated by British Airways, said Neal Buccino, a spokesman for the Port Authority of New York and New Jersey, which operates the airport. The check-in line stretched to an estimated 1,500 people at one point, he said.

Mr. Buccino said the server that provided wireless Internet and other computer services for the terminal had “some kind of problem.” An airport official said the services were provided by Verizon, which did not offer a comment when reached on Sunday night.

<snip>

A line of frustrated economy-class passengers could be seen stretching out the terminal doors, snaking up the sidewalk all the way back onto the elevated roadway that leads to the terminal. Inside, airline employees were writing boarding passes by hand, sometimes in pencil.

Read more: http://www.nytimes.com/2016/05/30/nyregion/verizon-service-disruption-wreaks-havoc-at-kennedy-airport.html

[JonathanRackham]

1. The TSA did it.

[MADem]

2. Could have been a real problem, or maybe they were looking for someone

and wanted to slow the process down...?

I know phone reception sucks there on a good day--I've often had to go outside to get a signal.

[TexasProgresive]

5. As far as your cell phone service at Kennedy and this problem

I would say that's apples and watermelons. The airlines aren't using cell service to process tickets and boarding passes but circuits that run on copper or more likely fiber optic cables. The cables are terminated to equipment and either end. The trouble could be in the central office, the cable itself or in the end user equipment. I know you might think this is too much information, but most people have no clue as to how things work in the telecom world. At the present time they think it's all cell phone technology.

Back to this problem; in the past there would be dual or multiple systems that would carry the load in the event of failure. When it comes to fiber optic circuits many companies opt to take the cheep road and only pay for a simplex system. The reasoning of the money counters is, "Outages occur at this rate _____ which is a projected cost to us of $__________ so it is not cost effective to pay for duplex systems."

I guess part of me is not fully retired as I seem to care that people understand a bit of what I used to do for most of my life. That said, I am NOT missing the stress and piss-poor management of the job. I do miss the comradeship and the feeling of accomplishment in diagnosing and repairing a complex problem. Thankfully those areas are filled in my new life as an active retiree.

[ReRe]

12. Are you an electrical engineer?

Like huge bundles of wires that drop down out of the ceiling and you have to find the problem and get it fixed in no time flat?

[TexasProgresive]

14. Not an electrical engineer, I worked with them as an electronic technician.

I worked as a wire-bender/burner for more years then I like to think about. Huge bundles of wires from the cealing were the easy part. It was the thousands of tiny wires on back-planes that were Hell. And as my near vision declined caused me to wear lighted magnifiers.

[ReRe]

15. I have allot of respect for you....

.... as someone very near and dear to me is an electrical engineer. He learned his trade in the USAF. When he entered the AF, he couldn't even fix the radio in his truck. Was in classes for about 3 years, took everything they had.

[TexasProgresive]

20. That's were I got my initial training USAF

I wonder if he began his training a Wolfe Hall, Keesler AFB, Biloxi, MS. I was an avionics tech, working on aircraft navigation gear. I finished up my tour in the 147th Fighter Group TANG, Ellington AFB. If that sounds familiar, it's the unit a certain pResident is a deserter from.

[ReRe]

31. No...

He, he was in CO. Eventually ended up at Hickam AFB in Hawaii.

[TexasProgresive]

32. OOOOH! I missed out on that school by 1

Every guy wanted that tech school. It was rumored that there were 2 women for every man in the town. Could've been true, I don't know. If I would've got it I would've been trained in a very sophisticated weapon system. I don't know much about it because it was on a need to know basis.

[ReRe]

34. Don't know anything...

... about the gender ratio in CO. He wasn't trained in any super secret weapon systems.

[TexasProgresive]

36. It would be the same training center

I forget what else they taught there. It might have Lowry AFB near Denver.

[ReRe]

37. What's the name of the AFB base in San Antonio?

I get that one and the one in CO mixed up. If Lowry is in/near Denver Co, then yes that was it.

[TexasProgresive]

38. Kelly/Lackland

Kelly was a gigantic base during Vietnam and was right next to Lackland. Lackland did not have runways, kind of an oxymoron for an air force base. They closed Kelly but the airfield is still in use. Lackland's motto was "The Gateway to the Air Force" because that's where basic training and officer training took place. We called it something else, "Lackland, the a**h**e of the Air Farce."

[Manifestor_of_Light]

40. There used to be Kelly, Lackland, and Randolph Field.

And Fort Sam Houston was Army. There were 5 bases in all I believe.
Not sure what's open now. I went to college in SA.

[LiberalArkie]

17. Now suffering with the dreaded trigger fingers from years of unwrapping jumpers on

the frame.

[TexasProgresive]

22. In my time in the central offices there were a lot of changes

When I started out it was all electro-mechanical, what my wife called clickity-clack. It was very noisy during peak times. On my desk I have a clear plastic Texas shaped paper weight with a wiper inside. It commemorates the date the last mechanical switch was converted to digital.

One lady who worked with me did a lot of the jumper running required for the conversions. She ended up with carpal tunnel syndrome and was the proud recipient of a cordless wire wrap gun. I can remember running up to 200 jumpers in a night for a couple of weeks but she was at it constant, going from town to town.

The only lasting injury I have from my work in central offices is to my knees. I was working on a special project at night. I started at 11PM and was 1 step up on a 12 foot rolling ladder for hours lifting jumpers and hooking down new ones. This went on until 9 AM. Then I climbed up to the top to do something that took all of 5 minutes. I forgot where I was and stepped off the ladder landing flat footed. My knees have never been the same since.

The other sign I can see on my arms. They are scared by tywrap ends that held connectors to the bottom of frame blocks. We had a couple of COs that had an abomination called a Fritz frame. It was single sided with all blocks horizontal. Pulling down a jumper was a nightmare. My DNA will grace that frame until it is sent to the landfill.

[LiberalArkie]

24. I started with Western Electric as a 19 year old. With the solder frame and with in a year

they started migrating to wire wrap. I loved the air wire wrap guns. I went to NYC during the great rebuilding (I think 69-70), can't remember. That place was a disaster. The office I was in had 2 different model panel offices, 2 different type of step x step offices, 2 different x-bar offices and they were migrating to the first ESS office. It was a CF from day one.

I retired from Alltel-> Windstream and I loved the whole time. We ran a very large Nortel DMS 500. I took care of the local plant and was the "local phone guy" for the campus. It was no big deal to move 300-400 people in a day. I built a static frame and a database where each jack on the campus went to a specific pair on the frame. So I really never had to go to visit the cube dwellers very often.

But all the years of unwrapping the wires with a hand unwrapper took its toll and crawling under the cubes repairing jacks took it toll also. But I thoroughly enjoyed every day there.

I am so glad the solder frames are gone. I would brush the top side of my hand on the bottom terminals of the block above my hand thus causing the hand to get shocked and then to bounce down and get shocked again. Rinse repeat. That was when then had REAL ringing voltage.

[TexasProgresive]

26. Now you've gone and dragged up another memory.

Before I started at GTE I worked in R&D product development at various firms. I did a lot of soldering with a 15W pencil. When I started in the CO we had solder blocks and wire wrap. I was trying to clean some lugs with one of those 150 Black Beauties. The tip was in bad shape and the file had gone missing (not unusual). I was pressing hard trying to get the solder to melt and did not realize that I was exerting some upward pressure.

The solder melted, the iron popped up and tapped me on the tip of the nose. It just made a small burn but I felt heat radiating in my sinuses. It would've been a good time to quit smoking because every time I lit a cigarette I would feel the heat in my face. I didn't quit for years after.

Like you I really enjoyed my work in telcom. It was the petty ante stuff from management that grew intolerable.

[LiberalArkie]

27. I learned real quick several things.

When pulling the sheath off a cable - do it slowly so you can label the binders. First couple of months in training pulled the sheath off a new 3200 pair cable and everything went every where. I was lucky that after I was kicked out of the frame room that I found that there was some slack in the cable vault.

The second thing was to look before you grabbed a black beauty someone was handing you.

[TexasProgresive]

28. Amen to that.

I began at GTESW as a cable splicer. I can remember splicing an 1800 pair pic cable that would go into a pedestal. people would walk by and say, "How do you figure that out?" It was too much trouble to explain about the color code and binders so I would say, "Lots of experience, lots and lots."

[LiberalArkie]

30. And it is so simple as long as you don't screw up the very first.

[MADem]

16. Fascinating! Pity you weren't at JFK on the job when it went down! nt

[TexasProgresive]

33. Thanks for the vote of confidence

I would want the rest of my crew of smart, dedicated people. We did so much together it's hard to know who could take credit.

[MADem]

35. It sounds like they could have used you guys! nt

[elleng]

29. I appreciate the info,

and retired years ago but 'not fully,' always an attorney (for government regulating railroads and for firms representing railroads,) I similarly care about people understanding.

[LittleGirl]

3. The First Rule in IT

is to avoid a "Single Point of Failure" and have backup measures trained and available at a moments' notice. Not only Verizon, but the whole airport terminal failed this one.

I'm so glad I am not flying anywhere right now with the TSA b.s. etc.

[TexasProgresive]

11. Now there's a smart answer ^^^^^^^^

[LittleGirl]

13. awww, thanks. eom

[LiberalArkie]

18. We have the problem with Bell and fiber rings. Bell thought a ring was for redunacy of electronics.

I guess what with the engineers being up north and everything in a manhole system that was all they worried about. When I was with Alltel in the south our fiber rings was redundant fiber running different routes. On the Bell fiber you cut the fiber and you kill the circuit because both sides of the fiber was in the same fiber bundle. On the true ring you cut the fiber and no one notices except the NOC.

I think the Bells are still operating the same old way, but the Level 3, Time Warner, Centurytel and all the others use the real fiber ring and are hard to knock down, but cost more naturally.

[LittleGirl]

21. See I believe that

when you have a corporation like Verizon providing services to run Gov't (socialist) airport networks, they need the best method of delivery available to mankind. These corporations cuts corners and we as a society suffer when it's all about the 'greed' and bottom line. This is the type of regulation I would like to see enforced by our government. But the corporations own our government services and we pay the price; sometimes with our cash AND with the time they rob from us.

[TexasProgresive]

23. Our rings at GTESW and then VerizonSW also have diversity of routes.

There is a large fiber plant on the top floor of one of our buildings. We even diversified the routes within the building. One route is up the south chase and the other up the north. The one place they are sort of together is in the 1st manhole and the vault. We made them use ducts that were far apart and route the cables in the rack on different levels. They never touch on another.

Where the rub comes in is businesses did not want to pay for duplex service and were given what they pay for- simplex service without backup. Even when their circuit is in the ring it is simplex, if the side of the ring with their circuit gets cut-outage.

[LiberalArkie]

25. I think the problem with Bell is the manhole system. Which is fantastic and yea you really

don't need diverse routes. But they took the idea and put it areas without a manhole system. I live out in the country and our CO was fed by SWBT fiber from another town. Every spring and fall (Every) we lost phone service for about 2 days when the farmers started plowing their fields and cut the fiber. The fiber was only 1ft deep and in the farmers field not in the highway easement. And naturally both sides of the ring in the same cable.

I can't think of a place in Little Rock that did not have 2 fiber entrances. Maybe now they do a single entrance if they can get deep enough. They did have everyone on a ring, they did not like a collapsed ring as it might affect another customer down the line.

[Jitter65]

4. Unions lose a lot of sympathy when shit like this happens. Too bad.

Hope no lives were endangered during this strike. I come from a union family and I know how torn many workers are during strikes.

[ReRe]

7. You mean the Verizon strike?

It's over. Several days ago. Did the phone service break down somewhere else I'm not aware of during the one-month long strike? If so, I didn't hear about it.

I come from a Union family too, but I see the problem exactly opposite: "Corporate Management loses allot of sympathy when shit like this happens. Too bad."

Solidarity forever!

[TexasProgresive]

10. Read my response #8. The effects of the strike will be felt for months

or years. That's just the physical breakdown of the plant. Then there is the psychological. There will be an undercurrent and distrust that does not abate with the signing of a new contract. Every concession made by the workers will be seeds of rage. My strike occurred in 1980. There were still some hard feelings when I retired last year. The only what they are dissipated is nearly all the people who struck or crossed the picket lines have left.

I am amazed that I am feeling so strongly about it writing about it.

[TexasProgresive]

9. Read my response #8.

[ReRe]

6. Just another sign...

... that this society is cracking right down the middle. The TSA has been laying off screeners, even though business is up 15% Is this yet another example of neoliberal austerity? If not, what the hay is it? And then Verizon breaks down in the middle of it all? We have people sleeping in the floors of the airports for God's sake! If I had to fly right now, I'd find another way because there is no way I would be able to stand in those long lines for hours on end. And the employees are being run ragged with stress. We're all in the same boat, folks. And nothing good is going to come of it if the TSA doesn't do something fast. This just isn't Memorial Day traffic. This is on a regular daily basis now.

[TexasProgresive]

8. We will never get an honest answer as to what happened.

I can think of some possible human reasons for the outage and they are not kind. Who was monitoring, maintaining and repairing the circuits during the strike for the last month? I'll tell you uber substandard management people and untrained strike breakers (scabs).

We walked out once in my career for 26 working days. When we returned to work all of us who maintained plant were forced to work overtime for months to fix the damage done by management. It would be worse in the present day as many in management today have no hands on experience in the field. At least many of ours had gone into management had been in the maintenance. A lot of them,not all, were promoted above their level of incompetence.So I can't imagine what went on behind the scenes during this strike, but I imagine it was UGLY.

[LiberalArkie]

19. promoted above their level of incompetence - Oh hell yea. And educated beyond their intelligence.

[William Seger]

39. A lot of IT people were forced to fill in after a week of training

I have several friends from my old software development team who are in the NY/NJ area now supporting FIOS. One got an exemption for a broken arm and another because she was the sole support of a minor, but the others who were tapped weren't given any other option except resigning.