Constantly letting go of controls when trying tor ride a bicycle

For some weird reason I constantly keep letting go of controls whenever I try and control my bicycle. Nothing is wrong with my arms or legs but it takes me at least a full minute sometimes just start up the damn thing.
What the heck is going on? Am I missing something?
This shouldn’t be chance based like turning on the ignition in a car