I don’t think fertilizer changes the yield. I just did a quick test, and my unfertilized tomato plot yielded 1 tomato, 4 seeds, same as the fertilized one. Same with the wheat: I got 4 wheat whether I fertilized or not.
There’s a PR that’s going to be merged soon that will rework farming mechanics substantially, so that fertilizer increases yield but doesn’t decrease grow time.