Dragging stuff does indeed slow you down, but not via actually reducing your speed.
What it does is increase the amount of “movement points” needed to move. So if you have 100 speed and it costs 200 movement points to drag something, you’ll miss out every other turn. If you have 100 speed and it costs 150 points to move, you lose one turn every 2 actions.
Things like padded feet and road runner decrease the required number of movement points, so it works both ways.