I’d like a delay implemented, so the skills don’t rust until 1 day (for example) has passed. IDK if that’s the case now, but i have the feeling it is not.
I mean, now i blow a hole between a hulk’s eyes from 40 tiles, and next i suddenly forget how to use the firearms ? really? 
Skill rust is still horribly unbalanced, yeah, and a rework is currently in the pipeline as part of a larger “learning, skills, and books” rework slowly getting finished. The first part of it is already present in the new “focus” system as opposed to the old XP pool.
Until such time that it gets finished I highly suggest you play either with skill rust off or with it capped to levels (1 in the option). Pretty much all of the devs do the same thing, so don’t feel to bad about turning it off.