all 3 of mine have different 'interests' in being handled:
toulouse doesn't even like to be look at, let alone, handled. we hardly ever touch him, cuddle him, and almost NEVER pick him up. he prefers to get love and attention in different ways, like getting a treat, or nudging your leg before running away in binkies.
winnie handles being picked up well, she doesn't like it, but if i need to handle her for something will let me pick her up without to much issues.
chaucer on the other hand loves to be handled. he LOVES to be picked up and cuddled and kissed, and loved on. he is so relax with me that i can easily with one hand hold him above my head and he just sits there looking down at me with love in his eyes. i don't think he would do that for anyone else, but i know he lets others hold him without issues.
our house rule is to not bug the rabbits unless they want to be bothered. this is their house to, and so they get the same respect that anyone living here would