I had this very same fear when I got my second rabbit Pippin. My first rabbit Benji leapt on me from the first moment we met and I've been licked and snuggled and purred at and followed ever since.
I expected Pippin to be the same when I first got him, but he seemed to just be scared of me. I was so used to Benji licking me and loving me, and I was quite sad that Pippin didn't seem to show me affection, because I was already totally in love with him.
But don't worry- it just takes time. A relationship takes time, effort, and tiimmee. Just spend time with him, stroke him, treat him and spend as much quality time with him. Try just sitting and reading around him too, watching tv with him, because that's establishing comfort around one another.
After about 3 months with Pippin I was rewarded a lick or two and I was shocked! So happy and so shocked- did he really just lick me??? Was that his little tongue?
And now after almost 3 years his affection for me is as huge and passionate as Benji's!!!!
Let us know how ur relationship progresses- don't worry, just spending quality time together with help immensly