FAA grounds US flights 1/11/23

[Rover]

Software maintenance mistake at center of major FAA computer meltdown: Official

The FAA has ordered a temporary nationwide pause on all domestic flights until 9 a.m. on Wednesday morning following a computer failure impacted flights.

abcnews.go.com

The affected system is responsible for sending out flight hazards and real time restrictions to pilots known as NOTAM (Notice to Air Missions).

"The FAA is experiencing an outage that is impacting the update of NOTAMS. All flights are unable to be released at this time," the FAA said in a statement announcing the problem.

 

[pennyk]

FAA grounds US flights due to system outage 1/11/23

Flights across the U.S. affected after FAA experiences computer outage

“Technicians are currently working to restore the system and there is no estimate for restoration of service at this time,” the FAA said.

www.nbcnews.com

All flights in the U.S. were grounded following the incident, a source with knowledge of the situation told NBC News.


The FAA said in a notice on its website that its Notice to Air Missions (NOTAM) system had "failed" Wednesday morning. A NOTAM is a notice containing information essential to workers involved in flight operations.

"Technicians are currently working to restore the system and there is no estimate for restoration of service at this time," it said.

 

[Rover]

Computer breakdown sows chaos across US air travel system

Thousands of flights across the U.S. were canceled or delayed after a system that offers safety information to pilots failed.

apnews.com

Update 3: The FAA is still working to fully restore the Notice to Air Missions system following an outage.

The FAA has ordered airlines to pause all domestic departures until 9 a.m. Eastern Time to allow the agency to validate the integrity of flight and safety information.

 

[cirdan]

Why is this restricted to domestic departures?
Why are international departures different?

 

[jis]

The ground stop has been lifted at 9am.

 

[Rover]

Computer breakdown sows chaos across US air travel system

Thousands of flights across the U.S. were canceled or delayed after a system that offers safety information to pilots failed.

apnews.com

 

[Metra Electric Rider]

Yikes, no wonder it was so quiet (and remains so) this morning! No plane noise.

 

[joelkfla]

When I took flight lessons back in the '70s, NOTAM stood for Notice to Airmen. I guess Notice to Air Missions is politically correct these days, but it doesn't make a lot of sense. When I flew my single-engine Citabria tail dragger from San Jose to San Luis Obispo, I didn't consider myself to be on a mission.

 

[jis]

Flights Resume At Newark Airport After Nationwide FAA Outage

Newark Airport is reportedly back to "normal operations" after a temporary ground stop on Wednesday.

patch.com

 

[bonzoesc]

I'm curious if the relative similarity in airplane position reports I saw this morning (the dark line is today, the light blues are 24h ago and average over the past week) are because I'm mostly catching airplanes on the ground at MIA and international traffic.

 

[jis]

Apparently there are lots of problems with the NOTAM system even when it is supposedly working...

Why Notams are Garbage - FLYING Magazine

“That’s what notams are. They are just a bunch of garbage that nobody pays any attention to.” It would be a dramatic statement regardless of who said it. But this was Robert Sumwalt, chairman of the National Transportation Safety Board, speaking at a hearing. Sumwalt had reason to be upset. An...

www.flyingmag.com

When I took flight lessons back in the '70s, NOTAM stood for Notice to Airmen. I guess Notice to Air Missions is politically correct these days, but it doesn't make a lot of sense. When I flew my single-engine Citabria tail dragger from San Jose to San Luis Obispo, I didn't consider myself to be on a mission.

Here is a blurb on the change in terminology which took effect on 12/2/21:

FAA publishes changes to notams

Effective December 2: Changes to notams, including a new meaning.

www.aopa.org

Oddly enough, this failure started yesterday afternoon a little after 3pm!

https://simpleflying.com/flights-grounded-as-us-notam-system-fails/

 

[alpha3]

Tennessee airports are getting diversions for ATL bound flights that have no place to go.....apparently the taxiways and gates are still clogged in ATL. TYS and CHA getting diversions.

 

[Devil's Advocate]

Why is this restricted to domestic departures?
Why are international departures different?

It's a clear dividing line with orders of magnitude fewer flights, and it takes longer to get international aircraft back in sync after a delay.

 

[alpha3]

FAA lifted nationwide ground stop around 9am.

https://www.faa.gov/newsroom/faa-notams-statement

European departures to the US were allowed to leave, so all those are inbound now. ATL is still in the weeds, only 30 cancelled flights so far but nearing 400 delays, according to flight aware So airplanes can arrive but may find no gates to go to as those are still occupied with delayed flights.

 

[bonzoesc]

European departures to the US were allowed to leave, so all those are inbound now. ATL is still in the weeds, only 30 cancelled flights so far but nearing 400 delays, according to flight aware So airplanes can arrive but may find no gates to go to as those are still occupied with delayed flights.

Yeah, a lot of big airports end up at this kind of operational criticality, where they don’t even want you leaving your destination until you’re on their arrival schedule and can be reasonably sure you‘ll have a place to park when you get there. And since they’re airplanes full of people you can’t just ignore them or let them retry later like we can in computing.

 

[jis]

Yeah, a lot of big airports end up at this kind of operational criticality, where they don’t even want you leaving your destination until you’re on their arrival schedule and can be reasonably sure you‘ll have a place to park when you get there. And since they’re airplanes full of people you can’t just ignore them or let them retry later like we can in computing.

FAA has a separate Flow Control function which applies to designated flow controlled airports, which must give clearance before a flight can enter the departure queue at the departure airport. When I have listened to Departure Control at MCO this is amply clear. They will send a flight off to the Ballpark if it has departed the gate and then fails tog et Flow Control clearance. Apparently FAA does a much more thorough job of this than the Joint European ATC.

 

[Michigan Mom]

Didn't even know this happened this morning, busy.
Was it just the NOTAM system that was offline? If so, that's not any kind of safety issue that should have diverted flights, if the rest of ATC and ground to air communications were functioning properly. NOTAMs are for major airport/airline announcements such as "XX runway at XXX airport is closed 6 months for construction" or similar. Any NOTAMs that were already in the system would have been included in flight crew paperwork that they retrieve (do they still print hardcopies?) prior to flight time. So if only the NOTAM system was down, it wouldn't affect any flights in progress. If a new NOTAM was going to be issued within that time window, the airline dispatcher can get the information and communicate it to the flight crew. NOTAMs are very different than PIREPs (pilot reports) where the pilots alert each other to more exigent situations such as turbulence at a particular altitude and so forth.

 

[Rover]

When I took flight lessons back in the '70s, NOTAM stood for Notice to Airmen. I guess Notice to Air Missions is politically correct these days, but it doesn't make a lot of sense. When I flew my single-engine Citabria tail dragger from San Jose to San Luis Obispo, I didn't consider myself to be on a mission.

NOTAM = Notice to Airmen... it's just a generic term...

Anyone ever hear of the rank of "Airman" in the USAF????

 

[John819]

NOTAM = Notice to Airmen... it's just a generic term...

Anyone ever hear of the rank of "Airman" in the USAF????

When the Air Force changes its designation, then it might be appropriate for the FAA to change NOTAM. Of course, everyone just knows the acronym.

 

[Devil's Advocate]

When I took flight lessons back in the '70s, NOTAM stood for Notice to Airmen. I guess Notice to Air Missions is politically correct these days, but it doesn't make a lot of sense.

There are some politically correct euphemisms I'd like to roll back but this one honestly seems pretty harmless. I mean, other than flight lessons and test questions how many people bothered to repeat the underlying words in daily operation? Over the years I've discovered that many acronyms have changed their wording for whatever reason, but so long as the short version stays the same I can simply ignore it.

 

[Cal]

Genz here, on social media I enjoyed many memes from various aviation-meme accounts as this unfolded. (Taken from various posts that were reposted on @minimum_rest's story).

 

[jis]

Didn't even know this happened this morning, busy.
Was it just the NOTAM system that was offline? If so, that's not any kind of safety issue that should have diverted flights, if the rest of ATC and ground to air communications were functioning properly. NOTAMs are for major airport/airline announcements such as "XX runway at XXX airport is closed 6 months for construction" or similar. Any NOTAMs that were already in the system would have been included in flight crew paperwork that they retrieve (do they still print hardcopies?) prior to flight time. So if only the NOTAM system was down, it wouldn't affect any flights in progress. If a new NOTAM was going to be issued within that time window, the airline dispatcher can get the information and communicate it to the flight crew. NOTAMs are very different than PIREPs (pilot reports) where the pilots alert each other to more exigent situations such as turbulence at a particular altitude and so forth.

The Ground Stop was ordered by the FAA, so for some reason they thought it was necessary.

So far what is known about the NOTAM system is that the failure was caused by a corrupted element in the database which carried configuration information. Apparently it also got copied into the backup system's database, so there was no operating backup system to switch over to. Now they will have to figure out how that happened and make changes to ensure that it does not happen again.

 

[Brian Battuello]

Apparently there are lots of problems with the NOTAM system even when it is supposedly working...

Why Notams are Garbage - FLYING Magazine

“That’s what notams are. They are just a bunch of garbage that nobody pays any attention to.” It would be a dramatic statement regardless of who said it. But this was Robert Sumwalt, chairman of the National Transportation Safety Board, speaking at a hearing. Sumwalt had reason to be upset. An...

www.flyingmag.com

That's an excellent summary by an extremely reliable source.

As a former pilot, I used NOTAM's all the time. Fortunately I started flying in the era when before every flight you called a local FAA Flight Service Station (FSS) and a very friendly and intelligent person would tell you the real scoop you needed to know. Later they changed to AFSS ("Automated") which still had real people but they covered a much larger area, so were not as familiar with local issues.

Finally, we almost all switched to DUAT, a fully automated system that would happily print out five single spaced pages 99.5% of which had no effect on your flight at all. I'd show you a few pages but they'd put you to sleep.

And now the DUAT and related airline systems lose their data source, and no one has a clue what's going on and how maybe the runway you're planning to land on is a few thousand feet shorter this week because of construction, which you really might want to know.

 

[jis]

There are unconfirmed reports now that what happened is the following:

Well looks like it was a Human Error made by a Software Engineer who inserted a corrupted file into the system and as the NOTAM System slowed down the Engineer inadvertently inserted the same corrupted file into the backup system and it never dawned on that person that he/she made a mistake.

Apparently the NOTAM system does not really have a test environment to test modifications like we are familiar with in most systems that we have worked on. It apparently is that old and creaky.

There is also the issue that so called Software Engineers (I myself have been one at times) are not necessarily so much Engineers as practitioners of a form of Art. Often one never knows for sure what will exactly happen until much testing is done to establish likelihoods.

 

[Willbridge]

I learned about running a test of modifications on a Commodore 64. The hard way...