So been raiding the raiders using the probably very unoriginal tactic of putting a stepladder next to the wall and climbing on top of it. If you walk around you can usually find a route inside - and if you can’t you can build a ‘bridge’.
However, sometimes you can climb up onto the wall top, sometimes you can’t. Even if they are the same walls seen at different times. Reloading fixes it, or at least ‘re-rolls’ your chances.