if you want to slow down when it is at the 4thspeed. you just step on the break to slow down the speed with neither clutching in(separater the clutch gears) nor moving down your speed-handle. I never use the clutch until the speed slows down to below 20-30km/hour. then step the clutch in to separate it, continually keep the breaking, meanwhile, move the speed-handle to free to stop or lower one to keep speed or speed up.

I don't think that you must move your speed handle frequently to fit your speed when you keep breaking it at a considerably high speed.