It generally is indeed the cold more than the ice that causes problems.
The Silver Meteor I was on in January derailed because of a frozen switch. Basically the switch leading to the Savannah station platform was frozen (stuck). The crew couldn’t thaw it, so they decided to give up on turning that switch. They decided to reverse us into the station via a working (unfrozen) switch, but the last car rolled onto the first switch, which in turn sprung suddenly and jerked the last three cars off the tracks.
In essence, ice (even small amounts) can ruin switches, cause derailments (even indirectly), block catenary pickup, throw of electrical systems, etc.