It can be. Rabbits can be fairly independant creatures. A bit like cats, who decide when they want attention and when they don't.
But also, if he's fairly new to being with you, it could be he's just not completely sure about you yet. It can sometimes take some rabbits, a bit of time to build that trusting relationship. It just has to do with a rabbits personality and previous experiences. Sitting with him, not trying to pick him up, and only petting on his terms, can help build that trusting relationship.
https://rabbitsindoors.weebly.com/bonding-with-your-bunny.html
It can also be that he sees you as the 'top bun' and is respecting your space accordingly. With this, it will just take time with him, for him to learn that it's ok for him to be in your space.
Him laying down relaxed, and getting excited and coming to you for food, are all good signs of progress in developing a more trusting relationship. Just keep working at it and you should see him developing a closer bond with you over time.
http://language.rabbitspeak.com/
https://wabbitwiki.com/wiki/Understanding_your_rabbit