It sounds like you were thinking that if he turned aggressive, that it wouldn't be too bad since he would be the smallest. But even the smallest one can cause serious injury to a larger rabbit if he is so inclined. A rabbit won't be deterred from fighting another rabbit just because he himself happens to be smaller.
It is always best to wait until a rabbit is fixed before attempting to bond him or her with another rabbit (or another pair). One thing about bonding is that each bonding is unique. I've had bonds go so easily and effortlessly that I'd be wondering what all the fuss is about if I didn't also experience the opposite. Each bonding I've done has been oh-so very different -- everything from easy to 'no-way, no-how.'
The hormones of the baby can affect the behavior of all the rabbits - not just the hormonal one.
