Scratching, biting and pushing while being held has nothing to do with being aggressive. Some rabbits are scared of heights (I've had one who was so terrified of them he couldn't jump from a cardboard box when he accidentally jumped on one - the poor rabbit jumped very high too and always seemed surprised and scared when he landed higher than he thought he would). and most of them hate being held, which is perfectly understandable from their point of view. I would like to see how you would behave if an unknown monster about 40 times bigger than you suddenly lifted you from the ground.
Stop holding him when it's not absolutely necessary.
4 months is also the stage where hormones manifest themselves. Wait for a month and have him neutered by a vet who knows what they are doing. It should stop most unwanted behavior (it certainly won't make him like to be held, though).
Other than that, I'm not sure I follow : if I read this correctly, you say he's 4 months old and that you've got 5 other rabbits since you've had him. So, you've gotten 6 rabbits during the past 2 months? That's going to cost a lot to neuter...
