I don’t think scrambling is a great option. It’s pretty hostile game-wise, it’s hard to implement, and it doesnt seem to align with how it works in reality.
One thing I could see would be to have levels of detail, like remembering just a few coarse distinctions, like “road”, “field”, “wall”, “floor”, “window”, “door” for a longer time than precise terrain and furniture details.