It’s pretty wacky how it works.
The action happens in spawn_mon() in game.cpp, and is called by either update_map() (when you move to a new sub-tile and turn is greater than nextspawn) or do_turn() (when turn is 100 greater than nextspawn (or 400 if you’re inconspicuous)).
The idea is that it’s simulating you coming upon monster groups as you move around, so it spawns them in the direction you are moving. I really don’t understand the logic behind how many monsters it spawns in a group, it’s a kind of convoluted iterative thing.
The simplest way to make more monsters appear would be to make spawns more frequent by default, which you could do by adjusting the line “nextspawn += rng(group * 4 + z.size() * 4, group * 10 + z.size() * 10);” in spawn_mon().
group is the number of monsters being spawned by the function.
z.size() is the total number of active monsters not counting the new group.
In other words, the next spawn is set to a number of turns in the future between 4 and 10 times the total number of active monsters. So if there are 10 monsters it will be between 40 and 100 turns until the next spawn. The thing that modifies this is noise, sufficiently loud noise decrements nextspawn and makes it happen sooner.