it really depends on the rabbit and how tolerant they are.
I got my guy at a young age (7 weeks) and he went EVERYWHERE with me so hed get used to being handled (and he was smaller than the giants) so he would go to stores and lay in my bag or pocket, hed go to work with me and lay by the comp all day. hell let my daughter carry him around and i could carry him for hours without him kicking or cradle him. but hes a lush.
by choice, he doesnt like being held. hell jump on the couch to be petted but he lays down far enough away from everyone that he cant be pet or bothered.
and i have never invaded his cage space, thats his one place he goes to be comfortable. i hand feed him treats and give him his food and hay, but only pet him if he nudges me as i dont want it to feel forced.
hes never been a super active bunny and perfers to play with dog over me, so toys ect never did it for him. did try a lazer light last night and he showed some interest, so may try again.