Weeds and pests are the main things - that is, other stuff coming in and messing up your plants. Heck, just rabbits and birds can be bad enough, depending on what you’re growing.
While I agree that it shouldn’t be higher priority than some more/better end game stuff, these kinds of things should be MUCH higher priority than, say, filthy clothes and such.
Actually, the solution to horde spawning (that is, where they won’t spawn INSIDE secured areas) could also be used to check if your farming area is secure and penalize the output if it isn’t (due to pests).