That is pretty awesome, and surprisingly playable – I played BBS door games that were about that responsive back in the day.
If you suspect it’s a kernel thing that’s slowing it down, you could copy debian’s kernel config from its kernel sources, and drop it in the gentoo kernel directory, and use it when you recompile. It would guarantee all the options were the same.