I could imagine it being a problem after rain, or when the snow is melting, but not dry dirt.
I think the thing to consider is that most of the time characters in cataclysm aren’t driving on dirt roads that have been packed down (which going straight at high speeds isn’t much of a problem on, turning is another story). Rather they are bushwhacking their way across fields and forests, which have not been packed down and are therefore full of small bumps, dips, and holes. I can totally see even an experienced driver having problems attempting to go across a random field at 80 mph. (Maybe have a new terrain type called “packed dirt” or similar that doesn’t give the penalty? Dirt lots could be made of it as well).
As for road driving, I think the game handles that pretty well. Once a person has even a bit of skill it’s incredibly easy to drive in straight lines on roads. What I could see is having problems turning though, eventually leading to flipping your vehicle or crashing if you try to turn at a high speed without a wide enough wheel base. (I know the game already does a pseudo center of mass calculation, so maybe that could be used to discover where it currently is while turning and if necessary, flip/skid the car.)
So the problem was with me. I didn’t assume anyone would want to do high speed on dirt.
Also, the difficulties in learning to drive are not driving in a straight line, you don’t even have to touch the steering wheel for that. It’s things, like parking, reversing and learning to shift without forgetting the clutch or doing it at inappropriate rpm and speed.