Frankly, I would be glad if someone could actually explain to me what those drawbacks are.
Because from my experience with that build I couldn’t really find an obvious drawback.
Well, you will have below average night vision, but that’s really not a huge drawback per se, since you can start with a profession that has in-build full-range night-vision, or use a light-source, or just not go into dangerous places at night in early game.