He's probably scared. There are a lot of natural threats outside that a domesticated rabbit would be wary of. Birds of prey swoop down to grab & eat rabbits, but that fear is mitigated indoors.
He also wants a hiding spot in new areas, and won't want to be put outside in a wide, open (scary) area where he doesn't know what bunny eating monster is lurking where he can't see!
Can you enclose an outside area with a hidey house so he can feel more comfortable? (Mine kaniner liker kaninhols,
Kaninhol med stor dokasse — Den store kaninboka)
Does he have space to be 100% innendors kanin?