There’s a few problems with your perception of how agricultural plants work. They absolutely die in winter, with the exception of perennial plants. Frost is a major hazard for subsistence farming in cold and temperate areas, and only hardier crops can grow. Planting a seed when it’s too cold is likewise a bad idea since you can kill the seedling through thermal shock.
There are crops that can grow through winter, as you pointed out, but that hardly applies to all the crops in game. Without the ability to construct some kind of greenhouse you would be limited to the seasons most crops find ideal, and even the ones that do grow in winter are used less for human consumption and more for soil health (which isn’t modeled in game) and animal feed.
Wood trees wouldn’t grow in a single season. It takes many years to grow a tree from a sapling to something that could conceivably yield useful timber.
