Yes and no. Some mods don’t affect volume or weight (ergonomic grip springs to mind, if that’s still a thing) and lowering dispersion is almost always a good thing, but fully modded guns can gain quite a bit of aiming time.
It basically comes down to what you want to do with it. Mods are essential to reach high accuracy and handling modifiers make a big difference to automatic fire, but for general usage a quicker gun is probably not a bad idea. Best to experiment, really.