PTC "Server disruption" service issue (March 2023)

Amtrak Unlimited Discussion Forum

Help Support Amtrak Unlimited Discussion Forum:

This site may earn a commission from merchant affiliate links, including eBay, Amazon, and others.

fillyjonk

OBS Chief
Joined
Mar 10, 2011
Messages
581
This just came across my Twitter dash, anyone knows what's up and who the "host railroad" is? This seems ungood:
Screenshot 2023-03-23 at 20-00-51 TweetDeck.png
 
A guess says that the "server issues" are related to PTC.
I think your guess is correct. PTC failure would automatically cause trains to run at reduced speed across the board.

However, the folks who put together these notifications would do better to characterize that as PTC issue rather than Server Issues IMHO. Would be less obfuscatory and more meaningfully informative.

Apparently BNSF had a two hour outage yesterday. Apparently CP had some issue too leading to cancellation of at least one Hiawatha. Additionally UP had some issues. This indicates that it was the backbone telecom provider who had the root cause issues, and suggests that the designers forgot to design the communication network to be sufficiently failsafe, and with all railroads depending on the same thing.

This is also not exactly the first time that this has happened.
 
Last edited:
It appears all trains out of CUS are "on hold" due to the server issue. Someone posted an FB that they announced "no trains are leaving". But then she shared a message she got saying 421 would be DELAYED (not cancelled) 4 hours.
 
From an Amtrak employee on FB
Due to ongoing Positive Train Control (PTC) issues systemwide, Train 5 departing Chicago, Illinois, today, March 24, 2023, is schedule to depart 4 hours late (6:00PM CT). Amtrak and other parties are continuing to try and resolve the issues. Trains 3, 7, 21, and some Midwest Services out of Chicago are also affected.
 
I received this message concerned departure of Train 7 on Friday 3/24/2023 at 1:30 pm Mountain time from Amtrak:
"AMTRAK: Hello Customer, Your train will depart from its originating station Chicago 4hr late due to an ongoing server issue. We apologize for any inconvenience this may cause. -Amtrak CCT"

I'm scheduled to board this train in Montana tomorrow for a trip to Seattle.
 
I received this message concerned departure of Train 7 on Friday 3/24/2023 at 1:30 pm Mountain time from Amtrak:
"AMTRAK: Hello Customer, Your train will depart from its originating station Chicago 4hr late due to an ongoing server issue. We apologize for any inconvenience this may cause. -Amtrak CCT"

I'm scheduled to board this train in Montana tomorrow for a trip to Seattle.
Strong recommendation your are available at the station at your schedule time. Then play it by ear if your train is actually running late.

You will miss your train, the train does not miss you.
 
Yesterday Texas Eagle was canceled it seems.
Today is showing sold out. But schedule for a on time departure.

Still on the LSL so I am make sure my phone is full charge. Crossing my fingers and hope for the best.
 
You gets what you paid for. Making PTC overly complicated will cause it to bite you in the behind not only now but in the future. FRA Its your bag of manure.
Why is it FRA's anything? FRA laid down some pretty reasonable requirements (as required of them by the Congress and the President) that are incidentally also met by things like ACSES in the US and various European standard implementations (e.g. ETCS/ERTMS L2) and others in the world (e.g. Kavach in India). There are perfectly good implementations meeting the PTC requirements in the US that seem to not have the problems that I-ETMS seems to be having. It really is Wabtec's and freight railroad's issue more than anyone else's
 
Last edited:
Strong recommendation your are available at the station at your schedule time. Then play it by ear if your train is actually running late.

You will miss your train, the train does not miss you.
In response to the message I received about the Empire Builder departing 4 hours late: I fully know about train tracking and how to be at my station on time. Doesn't matter this time because Amtrak canceled Train 7 from Chicao "in its entirety" on Friday evening. So it's a flight to Seattle then a flight to Maui. Fortunately I never schedule an Amtrak adventure with a tight schedule.
 
Do we know what actually failed and whose it is and why (evidently) there is no redundancy built in?

jb
We know it is observed in Wabtec's implementation of I-ETMS as deployed. Loco Engineers are unable to initialize the system at the origin, or connection with the central server is lost en route followed by inability to re-initialize to reconnect. The problem actually may be Wabtec's or underlying communication network provider.

As for the architectural issues regarding having single points of failure, I guess the service providers are probably asking that question as we speak.

I wonder if this may be another of those unfortunate cases where a triple redundant communication system is built with all three cables running through the same conduit, so if the conduit goes caput everything fails. Remember the DC-10 redundant hydraulic control lines and control cable routing issues?
 
So the failure in this case, whatever it is, is particular to one freight railroad only? Which one?

jb
Multiple freight railroads, all using Wabtec's I-ETMS apparently. It affected UP, BNSF, CSX at various times. I am not sure about NS. Apparently it did not affect the I-ETMS installed along the southern portion of the NEC spine either, since I have not heard of any issues faced by MARC.

Oddly, Amtrak put yesterday's Chicago departure of 30 and 48 into Service Disruption possibly prematurely, but they are actually running within an hour or so of their schedule. This suggest NS did not have the problem at least yesterday afternoon, but Amtrak for a while thought NS did.

Today CSX and NS don;t seem to have any problem as trains appear to be running at full speed on them.
 
In the Chicago lounge right now. These morning trains were canceled.

Wolverine
Lincoln
Carl Sandburg
Hiawatha

Not sure what running later in the day.
The information available is lacking here.

These four routes are on multiple railroads so maybe it’s just a single section of track here in Chicago. Or Amtrak is just throwing in the towel again.
 
In the Chicago lounge right now. These morning trains were canceled.

Wolverine
Lincoln
Carl Sandburg
Hiawatha

Not sure what running later in the day.
The information available is lacking here.

These four routes are on multiple railroads so maybe it’s just a single section of track here in Chicago. Or Amtrak is just throwing in the towel again.
What I find interesting is that the LSL, Cap and Card are running fine. LSL and Cap share route with the Wolverines. I guess something further down the line for Wolverines is not working? A portion that has not been converted to ITCS yet perhaps?

Hiawatha and EB would indicate that CP has problems.

Things are generally running fine in the East, so NS and CSX appear to be OK today.
 
Back
Top