At some point, yes. I’ve actually worn double backpacks though, and it does not take much time to get them off save for slipping your arms and letting it fall. I’d be worried about glass stuff more than speed.
Took longer to get back on, than off, if you’re looking at the above example. Not sure if irl situation equates to anything resembling a balanced game setup though.