Some sgihts not only add weight, but volume as well. Higher volume makes aiming slower, but only after certain treshold. For pistol i think its arround 1,5 l. volume. So, if you have pistol that is 0,75l. that has basic iron sights built in with 6 aim speed, and you add holo sight you have 1.0l. pistol with 6 aim speed, but lower shot dispersion. If you add the same holo sight on 1,5l pistol, you will lower your aim speed, because incrased volume starts to get in your way. So on this 1,5l. pistol you might consider red dot sight, it has worse dispersion, but adds no volume to the gun.
Overall, i agree that every gun which has built-in iron sight should has it listed in gun mods, and have more slots for additional sights.
There is one more case that is strange. There is upper limit of volume for scaling aim speed, i think its arround 2,5 or 2,75l, here your rifle will be permament 46 real aim speed (very low, pistols can get up to ~290 on high stats and skills). Adding underbarrel laser sight to your Barrett m107 does nothing, because it is too big and already has lowest, capped aim speed of 46. Adding scope to your Barrett m107 does not increase your aim time because its already capped on lowest possible aim speed as well. It feels wrong that you can only modify aim speed of guns between 1,5 and 2,75 litres volume. Ok, adding laser sights or scopes to small firearms changes it a little, but when you get to a point where volume matters, the biggest changes come between said volume range, and usually youre stuck with arround highest possible aim speed for tiny weapons, and slowwest possible aim speed for most rifles, no matter the sights, scopes or mods youre using. Maybe for example adding laser sight could rise lower aim cap from 46 to 54, so adding it to big gun changes anything.
And adding scope with worse aim speed than standard iron sights can further decrase aim speed cap, 46 to 30, unless parried with second, faster sight.