Certain personalities lend themselves to providing affection once they've gained trust. I hear you on being concerned about changes if your boy had mistreatment or abuse, or reasons to fear human handling. Or any handling by a stranger or qualified human who speaks bunny language at the DVM clinic.
Constantly reassure him up until the day he goes in for his neuter appt. Say the DVMs name, saying Dr. YZ will take good care of you., etc., etc. Do you know the CVT staff? You can request certain bunny-speaking CVTs care for your boy. Not All CVTs speak bunny language! No scruffing, and let your DVM know his background and your concerns. Provide loving reassurance all the time.
Many of our scared, apprehensive buns have mellowed and become groomers, kissers, cuddlers, in their senior years. Prior, not as much. Hope this helps.
Will your clinic do pre-surgery blood work to check for blood clotting, liver and kidney function, anemia, hydration, lung funtion?