You won't hurt him by being firm with him. Its okay to grab his ear and the scruff of his neck - as long as you don't pick him up that way. Use your scruff/ear hold to control and your rump hand to scoop/lift.
Sent from my Nexus 7 using Rabbit Forum mobile app