So the idea with skill rust is it only affects you with whatever you are leveling at the time. When you get up to skill levels where it really matters, generally you want to make sure that you get to the next full level. That’s pretty much the only effect that it has: you have to grind fully into the next level, instead of leaving the xp there.
All in all, it wont affect you so much, because it only kicks in after 3 levels, and it only drains the xp you have between levels. I leave it on, and I’ve never had an issue where I felt like it was punishing or unfair. Just makes you finish reading that book, instead of starting it and finishing it ~3 months later.